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CHAPTER  1 


Air  Traffic  Selection  and  Training  (AT-SAT)  Project 
Robert  A.  Ramos,  HumRRO 


INTRODUCTION 

This  document  is  a  comprehensive  report  on  a  large- 
scale  research  project  to  develop  and  validate  a  comput¬ 
erized  selection  battery  to  hire  Air  Traffic  Control 
Specialists  (ATCSs)  for  the  Federal  Aviation  Adminis¬ 
tration  (FAA).  The  purpose  of  this  report  is  to  docu¬ 
ment  the  validity  of  the  Air  Traffic  Selection  and 
Training  (AT-SAT)  battery  according  to  legal  and  pro¬ 
fessional  guidelines.  The  Dictionary  of  Occupational 
Titles  lists  the  Air  Traffic  Control  Specialist  Tower  as 
number  193162018. 

Background 

The  ATCS  position  is  unique  in  several  respects.  On 
the  one  hand,  it  is  a  critically  important  position  at  the 
center  of  efforts  to  maintain  air  safety  and  efficiency  of 
aircraft  movement.  The  main  purpose  of  the  ATCS  job 
is  to  maintain  a  proper  level  of  separation  between 
airplanes.  Separation  errors  may  lead  to  situations  that 
could  result  in  a  terrible  loss  of  life  and  property.  Given 
the  consequences  associated  with  poor  job  performance 
of  ATCSs,  there  is  great  concern  on  the  part  of  the  FAA 
to  hire  and  train  individuals  so  that  air  traffic  can  be 
managed  safely  and  efficiently.  On  the  other  hand,  the 
combination  of  skills  and  abilities  required  for  profi¬ 
ciency  in  the  position  is  not  generally  prevalent  in  the 
labor  force.  Because  of  these  chatacteristics,  ATCSs 
have  been  the  focus  of  a  great  deal  of  selection  and 
training  research  over  the  years. 

Historical  events  have  played  a  major  role  in  explain¬ 
ing  the  present  condition  of  staffing,  selection  and 
training  systems  for  ATCSs.  In  1981,  President  Ronald 
Reagan  fired  striking  ATCSs.  Approximately  1 1,000  of 
17,000  ATCSs  were  lost  during  the  strike.  Individuals 
hired  from  August  1981  to  about  the  end  of  1984 
replaced  most  of  the  strikers.  A  moderate  level  of  new 
hires  was  added  through  the  late  1980s.  However, 
relatively  few  ATCSs  have  been  hired  in  recent  years  due 
to  the  sufficiency  of  the  controller  workforce.  Rehired 
controllers  and  graduates  of  college  and  university  avia¬ 
tion  training  programs  have  filled  most  open  positions. 


Starting  in  fiscal  year  2005,  a  number  of  the  post- 1981 
hires  will  start  to  reach  retirement  eligibility.  As  a 
consequence,  there  is  a  need  for  the  Air  Traffic  Service 
to  hire  five  to  eight  hundred  ATCSs  candidates  a  year 
for  the  next  several  years  to  maintain  proper  staffing 
levels.  The  majority  of  the  new  hires  will  have  little 
background  in  ATCSs  work.  Further,  it  generally  takes 
two  to  four  years  to  bring  ATCS  developmentals  to  the 
full  performance  level  (FPL). 

In  addition,  the  FAA  Air  Traffic  Training  Program 
has  designed  major  changes  in  the  staffing  and  training 
of  new  recruits  for  the  ATCS  position.  In  the  past, 
training  at  the  FAA  Academy  included  aspects  of  a 
screening  program.  The  newly  developed  AT-SAT  se¬ 
lection  battery  is  designed  to  provide  the  vehicle  that 
will  screen  all  candidates  into  the  new  Multi-Path 
Training  Model.  One  of  the  important  characteristics 
of  the  new  training  process  is  that  it  will  no  longer  have 
a  screening  goal.  The  program  will  assume  that  candi¬ 
dates  have  the  basic  skills  needed  to  perform  the  work  of 
the  ATCS.  To  implement  the  new  training  model,  a 
selection  process  that  screens  candidates  for  the  critical 
skills  needed  to  perform  the  job  is  required.  A  Multi- 
path  hiring  model  implemented  with  AT-SAT  and 
augmented  by  a  revised  training  program  will  likely 
reduce  ATCS  training  length  and  time  to  certification. 

Given  this  background,  i.e.,  the  demographics  re¬ 
lated  to  potential  retirements,  and  new  staffing  require¬ 
ments  associated  with  training,  there  was  a  need  to  start 
the  ATCS  recruiting,  selection,  and  training  process  in 
fiscal  year  1997-1998.  In  spite  ofthis  immediate  need  to 
hire  recruits,  there  were  no  currently  feasible  selection 
processes  available  to  the  FAA  for  use  in  the  identifica¬ 
tion  and  selection  of  ATCSs.  Test  batteries  that  had 
been  used  in  the  past  had  become  compromised,  obso¬ 
lete,  or  were  removed  from  use  for  other  reasons. 

A  two-stage  selection  process  consisting  of  an  OPM 
test  battery  and  a  nine-week  Academy  screen  was  intro¬ 
duced  during  the  1980s  to  select  candidates  for  the 
position  of  air  traffic  controller.  This  two-stage  process 
was  both  expensive  and  inefficient.  First,  candidates 
took  a  paper-and-pencil  test  administered  by  the  Office 
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of  Personnel  Management  (OPM).  A  rank-ordered  list 
of  candidates  based  on  the  OPM  test  scores  was  estab¬ 
lished.  Candidates  were  listed  according  to  their  OPM 
test  score  plus  any  veteran’s  points.  Candidates  at  the 
top  of  the  list  were  hired,  provided  they  could  pass 
medical  and  security  screening. 

Once  candidates  were  hired,  they  entered  a  nine- 
week  screening  program  at  the  FAA  Academy.  Although 
modified  several  times  during  the  1980s,  the  basic 
program  consisted  of  time  spent  in  a  classroom  environ¬ 
ment  followed  by  work  in  laboratory-based,  non-radar 
simulations.  The  classroom  phase  instructed  employees 
on  aircraft  characteristics,  principles  of  flight,  the  Na¬ 
tional  Airspace  System,  and  basic  rules  for  separating 
aircraft  in  a  non-radar  situation.  During  the  ATCS 
simulations  phase,  employees  were  taught  and  evalu¬ 
ated  in  an  environment  that  emulated  the  work  per¬ 
formed  in  an  ATCS  facility. 

The  OPM  test  had  been  in  use,  without  revision, 
since  1 98 1 .  In  addition,  test  taking  strategies  and  coach¬ 
ing  programs  offered  by  private  companies  increased  the 
test  scores  of  candidates  without  an  apparent  compa¬ 
rable  increase  in  the  abilities  required  to  perform  in  the 
screen.  The  artificial  increase  in  test  scores  apparently 
reduced  the  capability  of  the  test  to  accurately  identify 
the  highest  qualified  individuals  to  hire.  Due  at  least  in 
part  to  the  artificially  inflated  OPM  scores,  less  than 
40%  of  the  ATCS  trainees  successfully  completed  the 
nine-week  screen.  A  full  discussion  of  prior  selection 
procedures  for  ATCSs  is  provided  in  Chapter  6  on 
Archival  Data  Analyses. 

Research  and  development  efforts  were  begun  to 
create  a  new  selection  device.  One  such  research  effort 
was  the  Separation  and  Control  Hiring  Assessment 
(SACHA)  project  initiated  in  1991.  SACHA  focused 
on  performing  a  job  analysis  of  the  air  traffic  controller 
position,  developing  ways  to  measure  ATCS  job  perfor¬ 
mance,  and  identifying  new  tests  suitable  for  selecting 
controllers.  The  SACHA  contract  expired  in  1 996. 

Another  research  and  development  effort,  the  Pre- 
Training  Screen  (PTS),  did  produce  a  one-week  selec¬ 
tion  test  designed  to  replace  the  nine-week  Academy 
screening  process.  However,  the  validity  of  the  PTS  was 
heavily  weighted  toward  supervisor  ratings  and  times  to 
complete  field  training,  along  with  performance  in  the 
Radar  Training  program.  The  FAA  continued  to  use  the 
PTS  to  screen  candidates  at  a  time  when  there  was  severe 
reduction  in  hiring,  but  there  was  no  near-term  poten¬ 


tial  to  be  hired.  Meanwhile,  the  SACHA  project  was 
already  underway  and  was  partly  envisioned  as  the  “next 
stage”  to  the  PTS. 

In  addition,  the  FAA  had  redesigned  the  initial 
qualification  training  program  in  anticipation  that  the 
PTS  would  filter  candidates  prior  to  their  arrival  at  the 
Academy.  The  nine-week-long  screening  process  was 
replaced  with  a  program  that  focused  on  training,  rather 
than  screening  candidates  for  ATCS  aptitudes.  As  a 
result,  the  FAA  had  an  initial  training  program  but  no 
pre-hire  selection  system  other  than  the  OPM  written  test. 

The  purpose  of  the  AT- SAT  project  was  to  develop 
a  job-related,  legally  defensible,  computerized  selection 
battery  for  ATCS’s  that  was  to  be  delivered  to  the  FAA 
on  October  1 , 1 997.  The  AT-SAT  project  was  initiated 
in  October  of  1 996.  The  requirement  to  complete  the 
project  within  a  year  was  dictated  by  the  perceived  need 
to  start  selecting  ATCS  candidates  in  1997. 

Organization  of  Report 

A  collaborative  team,  made  up  of  several  contractors 
and  FAA  employees,  completed  the  AT-SAT  project 
and  this  report.  Team  members  included  individuals 
from  the  Air  T  raffle  Division  of  the  FAA  Academy  and 
Civil  Aeromedical  Institute  (CAMI)  of  the  FAA,  Cali¬ 
ber,  Personnel  Decisions  Research  Institutes  (PDRI), 
RGl,  and  the  Human  Resources  Research  Organization 
(HumRRO).  The  Air  Traffic  Division  represented  the 
FAA  management  team,  in  addition  to  contributing  to 
predictor  and  criterion  development.  CAMI  contrib¬ 
uted  to  the  design  and  development  of  the  job  perfor¬ 
mance  measures.  Caliber  was  the  prime  contractor  and 
was  responsible  for  operational  data  collection  activities 
and  job  analysis  research.  PDRI  was  responsible  for 
research  and  development  efforts  associated  with  the  job 
performance  measures  and  development  of  the  Experi¬ 
ence  Questionnaire  (EQ).  RGI  was  responsible  for 
developmental  activities  associated  with  the  Letter  Fac¬ 
tories  Test  and  several  other  predictors.  HumRRO  had 
responsibility  for  project  management,  predictor  devel¬ 
opment,  data  base  development,  validity  data  analysis, 
and  tbe  final  report. 

The  final  report  consists  of  six  chapters,  with  each 
chapter  written  in  whole  or  part  by  the  individuals 
responsible  for  performing  the  work.  Contents  of  each 
chapter  is  summarized  below: 
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Chapter  1  -  Introduction;  contains  an  overview  of 
the  project,  including  background  and  setting  of  the 
problem  addressed,  and  methodology  used  to  validate 
predictor  measures. 

Chapter  2  -  Job  Analysis:  summarizes  several  job 
analyses  that  identified  the  tasks,  knowledges,  skills,  and 
abilities  required  to  perform  the  ATCS  job.  This  chap¬ 
ter  also  contains  a  linkage  analysis  performed  to  deter¬ 
mine  the  relationship  between  worker  requirements 
identified  in  the  job  analysis  to  the  predictor  measures 
used  in  the  validation  study. 

Chapter  3  -  Predictor  Development;  focuses  on  how 
the  initial  computerized  test  battery  was  developed  from 
job  analysis  and  other  information.  This  chapter  also 
discusses  construction  of  multi-aptitude  tests  and  alter¬ 
native  predictors  used  to  measure  several  unique  worker 
requirements  as  well  as  the  initial  trial  of  the  tests  in  a 
sample  of  students  in  a  naval  training  school. 

Chapter  4  -  Criterion  Development:  discusses  the 
development  and  construct  validity  of  three  criterion 
measures  used  to  evaluate  ATCS  job  performance. 

Chapter  5  -  Validation  of  Predictors:  presents  the 
predictor-criterion  relationships,  fairness  analyses,  and 
a  review  of  the  individual  elements  considered  in  decid¬ 
ing  on  a  final  test  battery. 

Chapter  6  -  Analyses  of  Archival  Data:  discusses  the 
results  of  analyses  of  historical  data  collected  and  main¬ 
tained  by  CAMI  and  its  relationship  to  AT-SAT  variables. 

Design  of  Validity  Study 
Step  1:  Complete  Predictor  Battery  Development 

The  tasks,  knowledges,  skills,  and  abilities  (worker 
requirements)  of  the  air  traffic  control  occupation  were 
identified  through  job  analysis.  Several  prototype  pre¬ 
dictor  tests  were  developed  to  cover  the  most  important 
worker  requirements  of  ATCSs.  The  management  team 
held  a  predictor  test  review  conference  in  November 
1 996  in  McLean,  Virginia.  At  the  meeting,  all  prototype 
predictor  tests  were  reviewed  to  determine  which  were 
appropriate  and  could  be  ready  for  formal  evaluation  in 
the  validity  study.  Twelve  individual  predictor  tests 
were  selected.  Step  1  of  the  management  plan  was  to 
complete  the  development  of  the  prototype  tests  and 
combine  them  into  a  battery  that  could  be  administered 
on  a  personal  computer.  This  initial  test  battery  was 
designated  the  Alpha  battery. 

It  was  also  decided  at  this  meeting  to  limit  the 
validation  effort  to  a  sample  of  full  performance  level  en 
route  ATCSs  to  help  ensure  that  the  validity  study 
would  be  completed  within  a  year.  Several  consider¬ 


ations  went  into  the  decision  to  perform  the  validation 
study  on  en  route  controllers.  Neither  the  development 
of  a  common  criterion  measure  nor  separate  criterion 
measures  for  en  route,  tracon,  and  tower  cab  controllers 
was  compatible  with  completing  the  validity  study 
within  one  year.  The  solution  was  to  limit  the  study  to 
the  single  en  route  specialty.  The  SACHA  job  analysis 
concluded  that  there  were  not  substantial  differences  in 
the  rankings  of  important  worker  requirements  between 
en  route,  tracon,  and  tower  cab  specialties.  In  addition, 
considerable  agreement  was  found  between  subactivity 
ratings  for  the  three  specialties.  Flight  service,  on  the 
other  hand,  appeared  to  have  a  different  pattern  of 
important  worker  requirements  and  subactivity  rankings 
than  the  other  options.  The  en  route  option  was  viewed 
as  reasonably  representative  of  options  that  control  air 
traffic,  i.e.,  en  route,  tracon,  and  tower  cab  specialists. 
Further,  the  number  of  en  route  specialists  was  large 
enough  to  meet  the  sample  size  requirements  of  the 
validation  study. 

In  addition,  Stepl  of  the  management  plan  included 
the  requirement  of  a  pilot  test  of  the  Alpha  battery  on  a 
sample  that  was  reasonably  representative  of  the  ATCS 
applicant  population.  Data  from  the  pilot  sample  would 
be  used  to  revise  the  Alpha  test  battery  on  the  basis  of  the 
results  of  analyses  of  item,  total  score,  and  test 
intercorrelations.  Beta,  the  revised  test  battery,  was  the 
battery  administered  to  en  route  ATCSs  in  the  concur¬ 
rent  validity  sample  and  pseudo  applicant  samples.  Test 
development  activities  associated  with  cognitive  and 
non-cognitive  predictors,  the  pilot  sample  description, 
results  of  the  pilot  sample  analyses,  and  resultant  recom¬ 
mendations  for  test  modifications  are  presented  in 
Chapter  Three. 

Step  2:  Complete  Criterion  Measure  Development 

Three  job  performance  measures  were  developed  to 
evaluate  en  route  job  performance.  By  examining  differ¬ 
ent  aspects  of  job  performance,  it  was  felt  that  a  more 
complete  measure  of  the  criterion  variance  would  be 
obtained.  The  three  measures  included  supervisor  and 
peer  ratings  of  typical  performance,  a  computerized  job 
sample,  and  a  high-fidelity  simulation  of  the  ATCS  job. 
Because  the  high-fidelity  simulation  provided  the  most 
realistic  environment  to  evaluate  controller  performance, 
it  was  used  to  evaluate  the  construct  validity  of  the  other 
two  criterion  measures.  The  research  and  development 
effort  associated  with  the  criterion  measures  is  presented 
in  Chapter  4. 
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Step  3:  Conduct  Concurrent  Validation  Study 

The  job  relatedness  of  the  AT-SAT  test  battery  was 
demonstrated  by  means  of  a  criterion-related  validity 
study.  By  employing  the  criterion-related  validation 
model,  we  were  able  to  demonstrate  a  high  positive 
correlation  between  test  scores  on  AT-SAT  and  the  job 
performance  of  a  large  sample  of  en  route  ATCSs. 
Because  of  the  amount  of  time  required  for  ATCSs  to 
reach  full  performance  level  status,  i.e.,  two  to  four 
years,  and  the  project  requirement  of  completion  within 
a  year,  a  concurrent  criterion-related  design  was  em¬ 
ployed  in  the  AT-SAT  study.  In  a  concurrent  validation 
strategy,  the  predictor  and  job  performance  measures 
are  collected  from  current  employees  at  approximately 
the  same  time. 

The  original  goal  for  the  number  of  total  participants 
in  the  study  was  750  en  route  ATCSs,  including  100 
representatives  from  each  of  the  major  protected  classes. 
Over  900  pairs  of  predictor  and  criterion  cases  were 
collected  in  Phase  I  of  the  concurrent  study.  However, 
the  goal  of  collecting  100  African  American  and  His¬ 
panic  ATCS  cases  was  not  achieved.  As  a  consequence, 
the  FAA  decided  to  continue  the  concurrent  validity 
study  to  obtain  a  greater  number  of  African  American 
and  Hispanic  study  participants.  These  additional  data 
were  required  to  improve  statistical  estimates  of  fairness 
of  the  AT-SAT  battery.  In  Phase  II,  data  were  collected 
from  en  route  sites  that  had  not  participated  in  Phase  I. 
In  addition,  a  second  request  for  study  participation  was 
made  to  ATCSs  in  sites  that  had  been  a  part  of  Phase  I. 
All  20  en  route  sites  participated  in  the  AT-SAT  study. 

It  should  be  noted  that  because  of  an  AT  CS  employee 
union  contract  provision,  each  study  participant  had  to 
volunteer  to  be  included  in  the  study.  Consequently,  the 
completion  of  the  study  was  totally  dependent  on  the 
good  will  of  the  ATCSs,  and  a  significant  amount  of 
effort  was  expended  in  convincing  them  of  the  need  and 
value  of  their  participation  in  the  AT-SAT  project.  A 
similar  effort  was  directed  at  employee  groups  repre¬ 
senting  the  protected  classes.  In  the  final  analysis,  how¬ 
ever,  each  study  participant  was  a  volunteer.  The  FAA 
had  no  real  control  over  the  composition  of  the  final 
ATCS  sample.  The  data  collection  effort  was  antici¬ 
pated  to  be  highly  intrusive  to  the  operations  of  en  route 
centers.  There  was  substantial  difficulty  associated  with 
scheduling  and  arranging  for  ATCS  participation.  The 
FAA’s  Air  Traffic  Training  management  team  had  the 
responsibility  to  coordinate  acceptance  and  participa¬ 
tion  in  the  study  of  all  stakeholders.  The  demographics 


of  the  obtained  samples,  corrected  and  uncorrected 
results  of  predictor  and  criterion  analyses,  and  group 
difference  and  fairness  analyses  are  discussed  in  Chapter  5. 

Step  4:  Conduct  Pseudo-Applicant  Study 

Full-performance-level  ATCSs  are  a  highly  selected 
group.  As  indicated  earlier,  even  after  the  OPM  selec¬ 
tion  battery  was  used  to  select  candidates  for  ATCS 
training,  there  was  still  a  40%  loss  of  trainees  through 
the  Academy  screen  and  another  1 0%  from  on-the-job 
training.  Under  these  conditions,  it  was  highly  likely 
that  the  range  of  test  scores  produced  by  current  AT  CSs 
would  be  restricted.  Range  restriction  in  predictor  scores 
suggests  that  ATCSs  would  demonstrate  a  lower  degree 
ofvariability  and  higher  mean  scores  than  an  unselected 
sample.  A  restricted  set  of  test  scores,  when  correlated 
with  job  performance  measures,  is  likely  to  under¬ 
estimate  the  true  validity  of  a  selection  battery.  There¬ 
fore,  to  obtain  validity  estimates  that  more  closely 
reflected  the  real  benefits  of  a  selection  battery  in  an 
unselected  applicant  population,  validity  coefficients 
were  corrected  for  range  restriction.  Range  restriction 
corrections  estimate  what  the  validity  estimates  would 
be  if  they  had  been  computed  on  an  unselected,  unre¬ 
stricted  applicant  sample. 

One  method  of  obtaining  the  data  required  to  per¬ 
form  the  range  restriction  corrections  is  to  obtain  a 
sample  of  test  scores  from  a  group  of  individuals  that  is 
reasonably  representative  of  the  ATCS  applicant  pool. 
The  best  sources  of  data  for  this  purpose  are  real 
applicants,  but  this  information  would  not  become 
available  until  the  test  battery  was  implemented.  Both 
military  and  civilian  pseudo-applicant  samples  were 
administered  the  AT-SAT  battery  for  the  purpose  of 
estimating  its  unrestricted  test  variance  and  correcting 
initial  validity  estimates  for  range  restriction.  Pseudo 
applicant  data  were  also  used  to  obtain  initial  estimates 
of  potential  differences  in  test  scores  due  to  race  and 
gender.  The  range  restriction  corrections  resulted  in 
moderate  to  large  improvements  in  estimates  of  validity 
for  the  cognitive  tests.  These  results  are  shown  in 
Chapter  5.  The  final  determination  of  these  corrections 
will,  by  definition,  require  analyses  of  applicant  data. 

Step  5:  Analyses  and  Validation  of  Predictors 

Data  management  was  a  particularly  critical  issue  on 
the  AT-SAT  project.  Plans  to  receive,  log  in,  and  process 
data  from  15  sites  ovet  an  eight-week  period  were 
created.  In  addition,  a  final  analysis  database  was  devel- 
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oped  so  that  the  validity  analyses  of  the  predictors  could 
be  completed  within  a  two-week  time  frame.  Plans  were 
also  made  on  the  methodology  used  to  determine  the 
validity  of  the  predictors.  These  included  predictor- 
criterion  relationships  and  reviews  of  the  individual 
elements  to  consider  when  deciding  on  the  final  test 
composite.  There  was  a  need  to  include  special  test 
composite  analyses  examining  the  interplay  of  differ¬ 
ences  between  groups,  the  optimization  of  particular 
criterion  variables,  and  coverage  of  worker  requirements 
and  their  effect  on  validity.  In  particular,  the  final 
composition  of  the  AT-SAT  battery  represented  a  com¬ 
bination  of  tests  with  the  highest  possible  relation  to  job 
performance  and  smallest  differences  between  protected 
classes.  All  validity  and  fairness  analyses  are  presented  in 
Chapter  5. 


Step  6:  Deliver  Predictor  Battery 
and  Supporting  Documentation 
The  final  deliverable  associated  with  the  AT-SAT 
project  was  the  AT-SAT  test  battery,  version  1.0,  on  a 
compact  disc  (CD).  The  goal  of  developing  a  selection 
test  battery  for  the  AT CS  that  was  highly  job  related  and 
fair  to  women  and  minorities  was  achieved.  Included 
with  the  CD  are  source  code,  documentation,  and  a 
user’s  manual.  In  addition,  a  database  containing  all  raw 
data  from  the  project  was  provided  to  the  FAA. 
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CHAPTER  2 


Air  Traffic  Controller  Job  Analysis 


Ray  A.  Morath,  Caliber  Associates 
Douglas  Quartetti,  HumRRO 
Anthony  Bayless,  Claudet  Archambault 
Caliber  Associates 


PRIOR  JOB  ANALYSES 

The  foundation  for  the  development  of  the  AT-SAT 
predictor  battery,  as  well  as  the  job  performance  mea¬ 
sures,  was  the  Separation  and  Control  Hiring  Assess¬ 
ment  (SACHA)  job  analysis  (Nickels,  Bobko,  Blair, 
Sands,  &  Tartak,  1995).  This  traditional,  task-based 
job  analysis  had  the  general  goals  of  (a)  supporting  the 
development  of  predictor  measures  to  be  used  in  future 
selection  instrumentation,  (b)  supporting  the  identifi¬ 
cation  of  performance  dimensions  for  use  in  future 
validation  efforts,  and  (c)  identifying  differences  in  the 
tasks  and  worker  requirements  (WRs;  knowledges,  skills, 
abilities,  and  other  characteristics,  or  KSAOs)  of  the 
different  ATCS  options  (Air  Route  Traffic  Control 
Center,  ARTCC;  Terminal,  and  Flight  Service)  and 
ATCS  job  assignments  (ARTCC,  Terminal  Radar  Ap¬ 
proach  Control,  TRACON;  Tower  Cab,  and  Auto¬ 
mated  Flight  Service  Station,  AFSS). 

Review  of  Existing  ATCS  Job  Analysis  Literature 
Nickels  et  al.  ( 1 996)  began  by  reviewing  and  consoli¬ 
dating  the  existing  ATCS  job  analysis  literature.  This 
integration  of  the  findings  from  previous  job-analytic 
research  served  as  the  initial  source  of  information  on 
ATCS  jobs  prior  to  any  on-site  investigations  and 
helped  to  focus  the  efforts  of  the  project  staff  conducting 
the  site  visits.  A  core  group  of  job  analysis  studies  also 
provided  much  of  the  information  that  went  into  devel¬ 
oping  preliminary  lists  of  tasks  and  WRs  for  the  SACHA 
project.  The  following  is  a  review  of  the  major  findings 
from  selected  studies  that  were  most  influential  to 
SACHA  and  provided  the  greatest  input  to  the  prelimi¬ 
nary  task  and  WR  lists. 


Computer  Technologies  Associates  (CTA) 

CTA  conducted  a  task  analysis  of  the  ARTCC, 
TRACON,  and  Tower  Cab  assignments  with  the  goal 
not  only  of  understanding  how  the  jobs  were  currently 
performed  but  also  of  anticipating  how  these  jobs  would 
be  performed  in  the  future  within  the  evolving  Ad¬ 
vanced  Automation  System  (AAS).’  They  sought  to 
identify  the  information  processing  tasks  of  ARTCC, 
TRACON,  and  Tower  Cab  controllers  in  order  to  help 
those  designing  the  AAS  to  gain  insight  into  controller 
behavioral  processes  (Ammerman  et  al.,  1983). 

An  extensive  assortment  of  documents  was  examined 
for  terms  suitable  to  the  knowledge  data  base,  including 
FAA,  military,  and  civilian  courses.  Listed  below  are  the 
sources  of  the  documents  examined  for  ATCS  terms 
descriptive  of  knowledge  topics  and  technical  concepts: 

•  Civilian  publications 

•  Community  college  aviation  program  materials 

•  Contractor  equipment  manuals 

•  FAA  Advisory  Circulars 

•  FAA  air  traffic  control  operations  concepts 

•  FAA  documents 

•  FAA  orders 

•  Local  facility  handbooks 

•  Local  facility  orders 

•  Local  facility  training  guides  and  programs 

•  NAS  configuration  management  documents 

•  National  Air  Traffic  Training  Program  (manuals, 
examinations,  lesson  plans,  guides,  reference  materials, 
workbooks,  etc.) 

•  Naval  Air  Technical  Training  Center  air  traffic  con¬ 
troller  training  documents 

•  U.S.  Air  Force  regulations  and  manuals 


*  Alexander,  Alley,  Ammerman,  Fairhurst,  Hosteder,  Jones,  &  Rainey,  1989;  Alexander,  Alley,  Ammerman,  Hosteder,  &  Jones, 
1988;  Alexander,  Ammerman,  Fairhurst,  Hosteder,  &  Jones,  1989;  Alley,  Ammerman,  Fairhurst,  Hosteder,  &  Jones,  1988; 
Ammerman,  Bergen,  Davies,  Hosteder,  Inman,  &  Jones,  1987;  Ammerman,  Fairhurst,  Hosteder,  &  Jones,  1989. 
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Detailed  task  statements  from  each  controller  option 
were  organized  hierarchically  into  more  global  and 
interpretahle  suhactivity  and  activity  categories.  Within 
CTA’s  framework,  one  or  more  tasks  comprised  a 
subactivity,  with  multiple  subactivities  subsumed  un¬ 
der  a  single  activity.  Hence,  this  approach  generated 
three  levels  of  job  performance  descriptors.  It  was  found 
that  controllers  in  each  of  the  three  ATCS  options 
(ARTCC,  TRACON,  and  Tower  Cab)  performed  about 
6-7  of  the  more  general  activities,  approximately  50  sub¬ 
activities,  and  typically  several  hundred  tasks. 

The  results  of  the  CTA  task  analysis  indicated  that 
activity  categories  as  well  as  subjectivity  categories  were 
similar  across  the  assignments  of  ARTCC,  TRACON, 
and  Tower  Cab,  with  only  small  variations  in  the  tasks 
across  the  options.  These  findings  suggested  that  the 
more  global  activities  performed  in  each  of  these  three 
controller  jobs  are  almost  identical.  Additionally,  job 
analysis  results  revealed  14  cognirively  oriented  worker 
requirements  (WRs)  that  were  found  to  be  critical  in  the 
performance  oftasks  across  the  three  assignments.  These 
WRs  were: 

•  Coding 

•  Decoding 

•  Deductive  reasoning 

•  Filtering 

•  Image/pattern  recognition 

•  Inductive  reasoning 

•  Long-term  memory 

■  Mathematical/probabilistic  reasoning 

•  Movement  detection 

•  Prioritizing 

■  Short-term  memory 

•  Spatial  scanning 

•  Verbal  filtering 

■  Visualization 

Human  Technologies,  Inc.  (HTI) 

HTI  (1991)  conducted  a  cognitive  task  analysis  with 
ARTCC  controllers  to  analyze  mental  models  and 
decision-making  strategies  of  expert  controllers.  An 
additional  goal  was  to  determine  the  effect  of  controller 
experience  and  expertise  on  differences  in  controller 
knowledges,  skills,  mental  models,  and  decision  strate¬ 
gies.  Cognitive  task  analysis  was  performed  by  videotap¬ 
ing  controllers  during  various  traffic  scenarios  and 
having  them  describe  in  detail  what  they  were  thinking 
while  they  were  handling  the  traffic  scenarios. 


The  general  findings  of  the  cognitive  task  analysis 
were  that  ARTCC  controllers’  mental  models  for  the 
control  of  air  traffic  could  be  broken  down  into  three 
general  categories,  which  were  termed  (a)  sector  man¬ 
agement,  (b)  prerequisite  information,  and  (c)  condi¬ 
tions.  These  three  categories  roughly  parallel  the 
respective  information  processing  requirements  of  short¬ 
term  memory,  long-term  memory,  and  switching  mecha¬ 
nisms.  The  resulting  12  cognitively  oriented  tasks  were: 

•  Maintain  situational  awareness 

•  Develop  and  revise  sector  control  plan 

•  Resolve  aircraft  conflict 

•  Reroute  aircraft 
■  Manage  arrivals 

•  Manage  departures 

•  Receive  handoff 

•  Receive  pointout 

•  Initiate  handoff 

•  Initiate  pointout 

•  Issue  advisory 

•  Issue  safety  alert 

Another  important  finding  from  the  study  was  that 
due  to  their  more  effective  working  memory,  experts 
have  access  to  more  information  than  novices.  That  is, 
experts  have  a  greater  chunking  capacity.  Experts  are 
also  more  efficient  in  the  control  of  aircraft  because  they 
typically  use  a  smaller  number  of  straregies  per  traffic 
scenario,  and  have  a  greater  number  of  strategies  that 
they  can  employ.  Finally,  the  study  found  that  expert 
ARTCC  controllers  differed  from  novices  in  their  per¬ 
formance  on  the  two  most  important  cognitive  tasks: 
maintaining  situational  awareness  and  revising  the  sec¬ 
tor  control  plan. 

HTI’s  work  also  involved  investigating  the  commu¬ 
nications  within  teams  of  radar  AT-SAT  (radar  and 
associate  radar)  as  well  as  the  communications  between 
radar  associates  and  controllers  in  other  sectors.  Teams 
were  studied  in  both  live  and  simulated  traffic  situa¬ 
tions.  Communication  data  were  coded  in  relation  to 
12  major  controller  tasks  that  were  found  in  the  cogni¬ 
tive  task  analysis.  The  data  indicated  that  nearly  all 
communication  between  team  members  concerned  the 
two  tasks  deemed  most  critical  by  the  cognitive  task 
analysis:  maintain  situational  awareness,  and  develop 
and  revise  sector  control  plan. 

A  summary  job  analysis  document  (HTI,  1993) 
presents  all  the  linkages  from  seven  sets  of  prototype 
documents  representative  of  air  traffic  controller  job 
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analyses.  The  objective  of  this  summary  process  was  to 
systematically  combine  the  results  from  the  air  traffic 
controller  job  analyses  into  a  single  document  that  empha¬ 
sizes  the  task-to-KSAO  linkages  for  the  jobs  of  En  Route, 
Fhght  Service  Station,  combined  TRACON  and  Tower 
(Terminal),  Tower,  and  TRACON  controllers. 

The  results  reported  in  the  HTI  summary  job  analy¬ 
sis  were  based  on  individual  job  analysis  summaries, 
which  included  a  cognitive  task  analysis  and  the  use  of 
the  Position  Analysis  Questionnaire  (Meecham  & 
McCormick,  1969).  The  HTI  analysis  also  utilized  the 
CTA  task  analysis  that  had  standardized  task  and  KSAO 
data  from  existing  air  traffic  control  job  analyses.  In  the 
individual  job  analyses,  the  controller  tasks  and  KSAO 
data  were  translated  into  Standard  Controller  Taxono¬ 
mies.  Then,  the  linkages  for  each  controller  job  were 
identified  and  placed  in  standard  matrices  based  on 
these  taxonomies. 

The  Task  Taxonomy  includes  a  total  of  4l  task 
categories  grouped  as  follows: 

■  Perceptual  Tasks 

■  Discrete  Motor  Tasks 

•  Continuous  Psychomotor  Tasks 

•  Cognitive  Tasks 

•  Communication  Tasks 

The  KSAO  Taxonomy  has  a  total  of  48  KSAO 
categories  divided  into  the  following  three  groupings: 

•  Abilities 

•  Knowledge 

•  Personality  Factors 

This  resulted  in  a  4l-by-48  task-to-KSAO  matrix 
that  permits  the  standard  listing  of  task-to-KSAO  link¬ 
ages  from  different  job  analyses. 

The  summary  document  (HTI,  1993),  which  incor¬ 
porated  the  individual  job  analyses  as  well  as  the  CTA 
report,  was  reviewed  and  utilized  by  the  AT-SAT  research¬ 
ers  in  determining  the  contribution  of  these  reports  to  the 
understanding  of  the  air  traffic  controller  job. 

Embry-Riddle 

Using  a  hierarchical  arrangement  of  activities  and 
tasks  borrowed  from  CTA,  Embry-Riddle  researchers 
(Gibb  et  al.,  1991)  found  that  five  activities  and  119 
tasks  subsumed  under  those  more  global  activities  were 
identified  as  critical  to  controller  performance  in  the 


non-radar  training  screen  utilized  with  ARTCC  and 
Terminal  option  controllers.  The  five  global  activities 
identified  by  Embry-Riddle  investigators  were: 

•  Setting  up  the  problem 

•  Problem  identification 

•  Problem  analysis 

-  Resolve  aircraft  conflicts 

•  Manage  air  traffic  sequences 

Upon  the  basis  of  the  results  of  the  task  inventory, 
existing  documentation,  and  the  information  obtained 
from  meetings  with  training  instructors,  the  following 
1 8  attributes  were  identified  as  critical  in  performing  the 
activities  and  tasks  involved  in  the  training: 

•  Spatial  visualization 

•  Mathematical  reasoning 

•  Prioritization 

•  Selective  attention 

•  Mental  rotation 

•  Multi-task  performance  (time  sharing) 

•  Abstract  reasoning 

•  Elapsed  time  estimation  and  awareness 

•  Working  memory  -  attention  capacity 

•  Working  memory  -  activation  capacity 

•  Spatial  orientation 

•  Decision  making  versus  inflexibility 

•  Time  sharing  -  logical  sequencing 

•  Vigilance 

•  Visual  spatial  scanning 

•  Time-distance  extrapolation 

•  Transformation 

•  Perceptual  speed 

In  addition  to  identifying  the  tasks  and  abilities 
required  for  success  in  training,  another  goal  of  this 
project  was  to  determine  the  abilities  necessary  for 
success  on  the  ATCS  job.  The  Embry-Riddle  team 
employed  Fleishman’s  ability  requirements  approach 
for  this  purpose.  Utilizing  the  task-based  results  of  the 
CTA  job  analysis  (Ammerman  et  al.,  1987),  they  had 
ARTCC,  TRACON,  and  Tower  Cab  controllers  rate 
CTA  tasks  on  the  levels  of  abilities  needed  to  success¬ 
fully  perform  those  CTA-generated  tasks.  Using 
Fleishman’s  abilities  requirements  taxonomy  (Fleishman 
&  Quaintance,  1984),  these  subject  matter  experts 
(SMEs)  rated  the  levels  of  perceptual-motor  and  cogni¬ 
tive  abilities  required  for  each  of  the  tasks.  It  was  found 
that  the  abilities  rated  by  controllers  as  critical  for 
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controller  performance  were  highly  similar  to  those 
found  by  the  CTA  study;  they  were  also  quite  similar  to 
those  abilities  identified  by  the  Embry-Riddle  team  as 
important  to  success  in  the  non-radat  training  screen. 
ARTCC  controllers  rated  the  following  abilities  from 
Fleishman’s  scales  as  necessary  to  perform  the  CTA- 
generated  ARTCC  tasks: 

•  Deductive  reasoning 

•  Inductive  reasoning 

•  Long-term  memory 

•  Visualization 

•  Speed  of  closure 

•  Time  sharing 

•  Flexibility  of  closure  (selective  attention) 

•  Category  flexibility 

•  Number  facility 

■  Information  ordering 

Those  abilities  rated  by  Terminal  controllers  as  re¬ 
quired  to  perform  the  Terminal  option  tasks  were: 

•  Selective  attention 

•  Time  sharing 

•  Problem  sensitivity 

•  All  of  Fleishman’s  physical  abilities  related  to  visual, 
auditory,  and  speech  qualities 

-  Oral  expression 
-Deductive  reasoning 

-  Inductive  reasoning 

-  Visualization 

-  Spatial  orientation 

-  All  perceptual  speed  abilities. 

The  Fmbry-Riddle  researchers  presented  no  discus¬ 
sion  on  why  differences  in  abilities  between  ARTCC 
and  Terminal  controllers  were  found. 

Landon 

Landon  (1991)  did  not  interview  SMFs,  observe 
controllers,  or  canvass  selected  groups  to  collect  job 
analysis  information.  Rather,  Landon  reviewed  existing 
documents  and  job  analysis  reports  and  summarized 
this  information.  Landon’s  focus  was  to  identify  and 
classify  the  types  of  tasks  performed  by  controllers. 
Using  CTA’s  hierarchical  categorization  of  tasks,  the 
ATCS  tasks  were  organized  into  three  categories  based 
upon  the  type  of  action  verb  within  each  task: 


I.  Information  Input  Tasks 
Receive,  interpret,  compare  and  filter  information 
Identify  information  needing  storage  or  further  pro¬ 
cessing 

A.  Scanning  and  monitoring 

B.  Searching 

II.  Processing  Tasks 

Organize,  represent,  process,  store,  and  access 
information 

A.  Analytical  planning 

B.  Maintain  picture  in  active  memory 

C.  Long-term  memory 

D.  System  regulation 

III.  Action/Output  Tasks 

Physical  and  verbal  actions  to  communicate  and 
record  information 

A.  Communicate  outgoing  messages 

B.  Update  flight  records 

C.  Operate  controls,  devices,  keys,  switches 

Myers  and  Manning 

Myers  and  Manning  (1988)  performed  a  task  analy¬ 
sis  of  the  Automated  Flight  Service  job  for  the  purpose 
of  developing  a  selection  instrument  for  use  with  AFSS. 
Using  the  CTA  hierarchy  to  organize  the  tasks  of  the 
job,  Myers  and  Manning  employed  SME  interviews 
and  surveys  to  identify  the  activities,  subactivities,  and 
tasks  of  the  job.  They  found  147  tasks,  21  subactivities, 
and  the  five  activities  that  they  felt  comprised  the  jobs  of  the 
AFSS  option.  The  activities  that  they  identified  were: 

•  Process  flight  plans 

•  Conduct  pilot  briefing 

•  Conduct  emergency  communications 

•  Process  data  communications 

•  Manage  position  resources 

Using  the  subactivities  as  their  focus,  Myers  and 
Manning  identified  those  WRs  required  to  successfully 
perform  each  subactivity.  Unlike  the  CTA  and  Embry- 
Riddle  job  analyses,  the  WRs  identified  were  much 
more  specific  in  nature,  as  evidenced  by  the  following 
examples: 

•  Ability  to  operate  radio/receive  phone  calls 

•  Ability  to  use  proper  phraseology 

•  Ability  to  keep  pilots  calm 

•  Ability  to  operate  Model  1  equipment 
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Summary  of  Previous  Studies 

SACHA  project  staff  summarized  the  findings  from 
the  previous  job  analyses  and  identified  the  commonali¬ 
ties  across  those  reports  regarding  the  tasks  and  worker 
requirements.  Specifically,  they  compared  CTA’s  worker 
requirements  with  those  reported  by  Embry-Riddle. 
Additionally,  once  the  SACHA-generated  lists  were 
completed,  the  researchers  mapped  those  worker  re¬ 
quirements  to  those  reported  by  CTA.  In  general,  the 
global  categories  of  tasks  and  the  hierarchical  organiza¬ 
tion  of  tasks  for  the  ARTCC  and  Terminal  options  were 
common  across  the  previous  studies.  Additionally,  the 
sub-activities  and  worker  requirements  identified  in 
previous  research  for  those  two  ATCS  options  were 
similar.  Finally,  the  previous  job  analyses  illustrated 
differences  in  tasks  and  worker  requirements  between 
the  AFSS  option  and  the  other  two  options. 

SACHA  Site  Visits 

After  reviewing  and  summarizing  the  existing  job 
analysis  information,  the  SACHA  project  staff  visited 
sites  to  observe  controllers  from  the  various  options  and 
assignments.  More  than  a  dozen  facilities,  ranging  from 
ARTCC,  Level  II  to  V  Tower  Cab  and  TRACON,  and 
AFSS  facilities,  were  visited.  The  primary  purpose  of 
these  initial  site  visits  was  to  gain  a  better  understanding 
of  the  ATCS  job.  SACITA  project  staff  not  only  ob¬ 
served  the  controllers  from  the  various  options  perform¬ 
ing  their  job,  but  they  also  discussed  the  various 
components  of  the  job  with  the  controllers,  their  train¬ 
ers,  and  supervisors. 

Development  of  Task  and  WR  Lists 

Developing  Preliminary  Lists 

On  the  basis  of  the  results  of  the  previous  job  analyses 
as  well  as  the  information  obtained  from  the  site  visits, 
SACHA’s  project  staff  developed  preliminary  task  and 
WR  lists.  Given  the  strengths  of  the  CTA  job  analysis 
regarding  (a)  its  level  of  specificity,  (b)  its  hierarchical 
arrangement  of  tasks,  and  (c)  its  focus  on  both  the 
current  ATCS  job  and  how  the  job  is  likely  to  be 
performed  in  the  future,  SACHA  decided  to  use  the  task 
analysis  results  of  CTA  as  the  basis  for  preliminary  task 
lists  with  the  options  of  ARTCC,  TRACON,  and 
Tower  Cab.  Similarly,  Myers  and  Manning  performed 
a  relatively  extensive  job  analysis  of  the  AFSS  position, 
which  had  been  modeled  after  CTA;  they,  too,  had  used 
a  hierarchical  categorization  of  tasks  and  had  a  high 
degree  of  specificity  at  the  molecular,  task  level.  Hence, 
four  preliminary  task  lists  were  developed  for  the  AT CS 


job  assignments  of  ARTCC,  TRACON,  Tower  Cab, 
and  AFSS,  with  task-based  findings  from  CTA  and  the 
Myers  and  Manning  results  serving  as  the  primary 
source  of  task-based  information  for  the  respective 
options.  Each  of  these  lists  contained  from  six  to  eight 
global  activities,  36  to  54  subactivities,  and  hundreds  of 
tasks. 

Several  steps  were  followed  in  the  development  of  the 
list  of  ATCS  worker  requirements.  On  the  basis  of  their 
review  of  existing  literature  and  their  knowledge  of 
those  relevant  KSAO  constructs,  SACHA  project  staff 
developed  an  initial  list  of  228  WRs.  They  then  con¬ 
ducted  a  three-day  workshop  dedicated  to  refining  this 
initial  list.  Consensus  judgments  were  used  to  eliminate 
WRs  that  were  thought  to  be  irrelevant  or  redundant. 
Finally,  a  single  preliminary  WR  list  was  formulated 
that  contained  73  WRs  grouped  into  14  categories 
(reasoning,  computational  ability,  communication,  at¬ 
tention,  memory,  metacognitive,  information  process¬ 
ing,  perceptual  abilities,  spatial  abilities,  interpersonal, 
work  and  effort,  stability/ adjustment,  self-efficacy,  and 
psychomotor).  This  list  of  WRs  is  presented  in  Table 
2. 1  and  will  henceforth  be  termed  the  SACHA-gener¬ 
ated  WRs  list. 

Panel  Review  of  Preliminary  Lists 

A  panel  of  five  controllers  assigned  to  FAA  Head¬ 
quarters  participated  in  a  workshop  to  review  and  revise 
the  SACHA  materials  and  procedures  for  use  in  addi¬ 
tional  job  analysis  site  visits.  These  five  controllers,  who 
represented  each  of  the  options  of  ARTCC,  Terminal, 
and  Flight  Service,  began  the  workshop  by  undergoing 
a  “dry  run”  of  the  planned  field  procedures  in  order  to 
critique  the  procedures  and  offer  suggestions  as  to  how 
they  might  be  improved.  Second,  the  panel  reviewed 
and  edited  the  existing  task  lists  for  the  various  options, 
mainly  to  consolidate  redundant  task  statements  and 
clarify  vague  statements.  Finally,  the  panel  reviewed  the 
SACHA-generated  WRs  list.  Upon  discussion,  the  panel 
made  no  substantive  changes  to  either  the  task  lists  for 
the  various  options  or  the  SACH  A-Generated  WRs  list. 

Developing  and  Revising  Task  Lists  in  the  Field 

In  field  job  analysis  meetings  held  in  different  re¬ 
gions,  the  preliminary  task  lists  were  presented  to  seven- 
nine  SMEs  (controllers  with  varying  levels  of  job 
experience)  from  each  of  the  four  options  (ARTCC, 
TRACON,  Tower  Cab  and  AFSS).  Special  attention 
was  placed  upon  having  groups  of  SMEs  who  were 
diverse  in  race/ ethnicity,  gender,  and  years  of  experience 
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controlling  traffic,  and  who  represented  various  levels  of 
ATC  facilities.  Attempts  were  made  to  avoid  mixing 
subordinates  and  their  supervisor(s)  in  the  same  meeting. 

Project  staff  instructed  SMEs  to  review  their  respec¬ 
tive  task  list  (whether  it  be  ARTCC,  TRACON,  Tower 
Cab  ,  or  AFSS)  to  confirm  which  tasks  were  part  of  their 
job  and  which  were  irrelevant.  In  addition,  SMEs  were 
asked  to  consolidate  redundant  tasks,  to  add  important 
tasks,  and  to  edit  any  task  statements  that  needed 
rewording  for  clarification  or  correction  of  terminol¬ 
ogy.  SMEs  proceeded  systematically,  reviewing  an  en¬ 
tire  group  of  tasks  under  a  subactivity,  discussing  the 
necessaiy  changes,  and  coming  to  a  consensus  regarding 
those  changes  before  moving  to  the  next  subactivity. 
After  editing  of  the  tasks  was  completed,  SMEs  were 
asked  to  identify  those  tasks  that  they  believed  were 
performed  by  all  ATCS  options,  as  well  as  those  tasks 
specific  to  one  or  more  ATCS  position  (s). 

These  meetings  produced  four  distinct  task  lists 
corresponding  to  ARTCC,  TRACON,  Tower  Cab, 
and  AFSS  controllers. 

Developing  SME-Generated  WRs  Lists 

SME  meetings  were  also  held  for  the  purpose  of 
having  controllers  generate  their  own  lists  of  WRs  that 
they  felt  were  necessary  for  effective  job  performance. 
SMEs  were  not  given  the  preliminary  WR  list  that  had 
been  generated  by  SACHA  job  analysts  but  were  in¬ 
structed  to  generate  their  own  list  of  skill  and  ability- 
related  WRs.  SMEs  were  utilized  to  identify  and  define 
WRs  while  project  staff  assisted  by  clarifying  the  differ¬ 
ences  and  similarities  among  the  WR  definitions.  That 
is,  project  staff  tried  to  facilitate  the  development  of  the 
WRs  without  influencing  the  SMEs’  judgments. 

As  a  result  of  this  effort,  SME  controllers  across  the 
three  options  of  ARTCC,  Terminal,  and  Flight  Service 
generated  47  WRs.  Based  upon  the  input  from  the 
SMEs,  definitions  were  written  for  each  of  the  WRs.  It 
was  concluded  that  all  47  SME-generated  WRs  would 
be  applicable  to  any  position  within  the  three  ATCS 
options.  Table  2.2  represents  the  list  of  47  SME- 
generated  WRs. 

When  the  SACHA-generated  and  SME-generated 
WR  lists  are  compared,  they  appear  to  be  quite  similar 
except  for  the  lack  of  metacognitive  and  information 
processing  WRs  in  the  SME-generated  list.  SACHA 
staff  reported  that  controllers  generating  the  SME  list  of 
WRs  lacked  familiarity  with  metacognitive  and  infor¬ 
mation  processing  constructs  and  were  probably  not  the 


best  sources  of  information  when  it  came  to  identifying 
these  types  ofWRs.  Because  (as  SACHA  researchers  stated) 
no  “common  language”  existed  with  which  to  discuss  these 
types  of  WRs,  project  staff  did  not  tty  to  pursue  defining 
these  types  of  WRs  with  the  controller  SMEs. 

Linking  Tasks  to  SME-Generated  WRs 

SME  meetings  were  then  held  to  have  controllers 
provide  linkage  judgments  (obtained  via  group  discus¬ 
sion)  relating  the  tasks  subsumed  under  a  particular 
subactivity  to  the  SME-generated  WRs  required  to 
perform  that  subactivity.  SMEs  from  the  ARTCC  and 
Terminal  options  reviewed  the  task  and  SME-generated 
WR  lists  from  their  respective  options  and  identified 
those  WRs  needed  to  perform  each  subactivity.  SMEs 
focused  upon  one  subactivity  at  a  time  and  obtained 
consensus  regarding  the  most  important  WRs  for  that 
subactivity  before  moving  on  to  the  next.  Linkages  were 
made  at  the  subactivity  level  because  the  latge  number 
of  tasks  precluded  linkages  being  made  at  the  task  level. 
Due  to  scheduling  problems,  the  SACHA  project  staff 
were  unable  to  hold  a  linkage  meeting  with  AFSS  SMEs, 
so  no  data  were  obtained  at  this  stage  linking  AFSS  tasks 
to  AFSS  WRs. 

While  two  SME-generated  WRs  (Motivation  and 
Commitment  to  the  Job)  were  not  linked  to  any  of  the 
subactivities,  controllers  stated  that  these  two  WRs  were 
related  to  the  overall  job.  Thus,  even  though  these  WRs 
could  not  be  directly  linked  to  the  tasks  of  any  specific 
subactivity,  controllers  felt  that  their  importance  to 
overall  job  performance  justified  the  linkage  of  these 
two  requirements  to  every  subactivity.  Results  of  the 
linkage  meetings  revealed  that  every  SME-generated 
WR  could  be  linked  to  at  least  one  subactivity  and  that 
each  subactivity  was  linked  to  at  least  one  WR. 

Developing  Consolidated  List  of  WRs 

At  this  stage,  SACHA  job  analysts  generated  a  con¬ 
solidated  WR  list  combining  the  SME  (controller)  and 
SACHA-generated  WRs.  They  began  the  consolidation 
process  by  including  45  of  the  47  original  SME-gener¬ 
ated  WRs.  Two  of  the  original  47  were  dropped  (Avia¬ 
tion  Science  Background  and  Geography)  because  they 
were  job  knowledges  rather  than  skills  or  abilities.  Next, 
the  SACHA-generated  list  of  WRs  was  reviewed  to  add 
WRs  that  had  not  been  identified  in  the  SME-generated 
list  but  were  considered  important  to  the  job  of  ATCS. 
Finally,  the  project  staff  added  two  WRs  (Recall  from 
Interruption  and  Translation  of  Uncertainty  into  Prob- 
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ability)  that  had  not  been  identified  in  either  the  SACHA 
or  the  controller  lists  but  were  deemed  necessary  to 
perform  the  job  from  job  analytic  suggestions. 

Thus,  a  final  consolidated  list  of  66  WRs  was  created 
(45  SME-generated  and  21  SACHA-generated)  that 
included  skills  and  abilities  in  the  areas  of  communica¬ 
tion,  computation,  memory,  metacognition,  reasoning, 
information  processing,  attention,  perceptual/spatial, 
interpersonal,  self-efficacy,  work  and  effort,  and  stabil¬ 
ity/adjustment  (Table  2.3). 

Job  Analysis  Survey 

Utilizing  the  information  gained  from  the  site  visits 
and  SME  meetings,  the  SACHA  staff  developed  a  job 
analysis  survey  and  disseminated  it  to  a  cross-section  of 
ATCSs  from  the  various  options  and  assignments  lo¬ 
cated  throughout  the  country.  The  main  goals  of  the 
mail-out  survey  were  to  identify  the  most  important 
WRs  for  predictor  development,  to  explore  criterion 
measures,  and  to  identify  possible  differences  in  the 
subactivities  being  performed  across  job  assignments. 
Of  the  1009  surveys  sent  out  to  ATCSs  in  February 
1994,  444  were  returned,  with  usable  data  obtained 
from  389  respondents. 

Content  of  the  Survey 

The  survey  was  divided  into  four  sections:  an  intro¬ 
duction,  a  subactivity  ratings  section,  a  Wffs  rating 
section,  and  a  background  information  section.  The 
introduction  explained  the  purpose  of  the  survey  to  the 
ATCSs,  provided  instructions  on  completing  the  sur¬ 
vey,  and  encouraged  participation.  The  background 
section  gathered  information  on  such  things  as  the 
respondents’  gender,  race,  job  experience,  facility  type 
and  facility  level. 

The  subactivity  rating  section  was  comprised  of  108 
entries,  the  combined  list  of  all  sub-activities  across  the 
ATCS  options  of  ARTCC,  Terminal,  and  Flight  Ser¬ 
vice.  The  instructions  informed  respondents  that  the 
survey  contained  subactivities  from  ARTCC, 
TRACON,  Tower  Cab,  and  AFSS  jobs,  and  thus  it 
would  be  unlikely  that  all  entries  would  be  relevant  for 
a  particular  job  assignment.  Respondents  were  asked  to 
rate  each  subactivity  on  (a)  its  importance  in  success¬ 
fully  performing  their  job,  and  (b)  the  time  spent 
performing  this  subactivity  relative  to  the  other  job 
duties  they  perform.  A  single  task  criticality  index  was 


also  created  by  combining  the  importance  and  relative 
time  spent  ratings.  This  index  provided  an  indication  of 
the  relative  criticality  of  each  subactivity  with  respect  to 
job  performance. 

The  WR  rating  section  of  the  survey  was  comprised 
of  67  items,  which  included  (a)  the  45  controller¬ 
generated  items,  (b)  the  two  SME-generated  job 
knowledges  that  had  been  reintroduced  into  the  survey, 
(c)  the  16  SACHA-generated  items  (five  of  the  21 
SACHA-generated  items  dealing  with  information  pro¬ 
cessing  were  left  off  the  survey  due  to  controllers’  lack  of 
understanding  and  familiarity  with  these  constructs), 
and  four  items  to  identify  random  responses.  Respon¬ 
dents  were  instructed  to  rate  each  of  the  67  WRs  on  both 
its  relative  importance  in  learning  the  job  and  its  relative 
importance  in  doing  the  job. 

Overview  of  Survey  Findings 

WRs.  Findings  revealed  very  little  difference  be¬ 
tween  the  WRs  seen  as  important  for  doing  the  job  and 
those  needed  to  learn  the  job.  Rank  orderings  of  the  WR 
mean  scores  for  doing  and  learning  the  job  were  highly 
similar.  This  result  appeared  to  hold  across  job  options 
and  job  assignments.  Mean  rankings  of  the  WRs  for  all 
ATCS  job  assignments  are  shown  in  Table  2.4.  These 
scores  reflect  the  mean  rankings  of  the  WRs  for  learning 
and  for  doing  the  job. 

The  results  also  suggested  that,  while  there  seemed  to 
be  no  substantial  difference  between  the  WR  ratings  of 
the  ARTCC  and  the  Terminal  option  controllers 
(TRACON  and  Tower  Cab),  the  Flight  Service  con¬ 
trollers  appeared  to  rate  the  WRs  differently.  They  rated 
WRs  dealing  with  information  collection  and  dissemi¬ 
nation  as  relatively  more  important  than  did  the  ART CC 
and  Terminal  option  controllers,  and  rated  WRs  deal¬ 
ing  with  metacognitive  functions  as  relatively  less  im¬ 
portant. 

As  a  result  of  the  findings,  SACHA  staff  felt  that  there 
were  no  substantive  differences  between  the  ARTCC 
and  the  Terminal  options  in  the  ordering  of  the  WRs, 
which  would  influence  predictor  development  for  these 
two  options.  However,  they  advised  that  any  future 
work  dealing  with  test  development  for  the  Flight 
Service  option  should  take  into  consideration  their 
different  rank  ordering  of  WRs.  Tables  2.5  and  2.6  list 
the  mean  ratings  of  the  WRs  (from  each  of  the  four  job 
assignments)  for  doing  and  learning  the  job. 
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Subactivities.  As  with  the  ratings  of  the  WRs,  the 
results  of  the  subactivity  ratings  revealed  that  ARTCC 
and  Terminal  option  controllers  shared  similar  profiles 
with  respect  to  the  relative  criticality  of  the  subactivities. 
While  the  ARTCC  and  Terminal  option  controllers 
share  more  common  subactivities  than  they  do  each 
share  with  the  Flight  Service  option,  1 1  subactivities 
were  given  relatively  high  ratings  by  all  three  options. 
These  common  sub-activities  were  associated  with  the 
safe  and  expeditious  flow  of  traffic,  as  well  as  responding 
to  emergencies  or  special  conditions  and  contingencies. 
Table  2.7  contains  the  ranked  mean  rating  of  the  sub¬ 
activities  across  all  ATCS  options. 

The  SACHA  staff  also  felt  that  another  important 
finding  from  the  controller  ratings  of  the  subactivities 
was  that,  regardless  of  job  option  or  assignment,  those 
subactivities  dealing  with  multitasking  were  consis¬ 
tently  seen  as  important  to  the  ATCS  job.  The  project 
staff  operationalized  multitasking  as  those  times  when 
controllers  must  (a)  perform  two  or  more  job  tasks 
simultaneously,  (b)  continue  job  tasks  despite  frequent 
interruptions,  and  (c)  use  multiple  sensory  modalities  to 
collect  information  simultaneously  or  near  simulta¬ 
neously.  When  dealing  with  the  ratings  of  the 
subactivities  across  all  ATCS  options,  it  was  found  that 
ten  of  the  1 1  sub-activities  dealing  with  multitasking 
had  criticality  scores  that  placed  them  in  the  top  third  of 
all  subactivities. 

Conclusions 

Considering  the  results  of  the  SACHA  job  analysis 
survey  and  taking  into  account  the  goals  of  this  selec¬ 
tion-oriented  job  analysis,  the  project  staff  arrived  at 
several  general  conclusions. 

•  There  appeared  to  be  no  substantial  differences  in  the 
rankings  of  the  important  WRs  between  ARTCC, 
TRACON,  and  Tower  Cab  controllers.  However,  the 
differences  in  the  rankings  found  between  Flight  Service 
option  controllers  and  the  other  options  did  appear  to 
be  substantive  enough  that  any  future  efforts  to  develop 
selection  instrumentation  should  take  these  differences 
into  account. 

•  Considerable  agreement  was  found  between  the 
subactivity  rankings  for  the  ARTCC,  TRACON,  and 
Tower  Cab  controllers,  while  the  rank  ordering  of  the 
subactivities  for  the  Flight  Service  option  appears  to  be 
different  from  all  other  options  and  job  assignments. 

•  Regardless  of  job  option  or  assignment,  multitasking 
is  an  important  component  of  the  ATCS  job. 


Linkage  of  Predictors  to  Work 
Requirements  Overview 

To  determine  whether  the  various  instruments  com¬ 
prising  the  AT-SAT  predictor  battery  were  indeed  mea¬ 
suring  the  most  important  WRs  found  in  the  SACHA 
job  analysis,  linkage  judgments  were  made  by  individu¬ 
als  familiar  with  the  particular  AT-SAT  measures,  as 
well  as  the  WRs  coming  out  of  SACHA.  Linkage 
analysis  data  were  collected  through  surveys.  Surveys 
contained  (a)  a  brief  introduction  describing  why  the 
linkage  of  tests  to  WRs  was  necessary,  (b)  a  brief 
background  questionnaire,  (c)  instructions  for  com¬ 
pleting  the  linkage  survey,  (d)  definitions  of  the  WRs 
from  SACHA’s  revised  consolidated  list,  and  (e)  linkage 
rating  scales  for  each  of  the  AT-SAT  measures.  Survey 
results  showed  that  each  of  the  measures  comprising  the 
AT-SAT  battery  was  successfully  capturing  at  least  one 
or  more  WRs  from  SACHA’s  revised  consolidated  list. 
Additionally,  the  vast  majority  of  those  WRs  being 
captured  by  AT-SAT  measures  were  those  SACHA 
found  to  be  most  important  for  both  learning  and  doing 
the  job. 

The  linkage  analysis  made  use  of  65  of  the  66  WRs 
from  SACHA’s  final  revised  consolidated  list.  Due  to  an 
oversight,  two  of  the  WRs  from  SACHA’s  list  were 
labeled  Rule  Application  (one  SME-generated  and  the 
other  SACHA-generated),  and  both  were  listed  under 
the  Information  Processing  category.  When  the  SACHA 
list  of  WRs  and  their  respective  definitions  were  being 
transcribed  for  use  in  the  linkage  analysis,  only  one  of 
the  two  Rule  Application  WRs  was  transcribed.  Hence, 
the  linkage  analysis  collected  linkage  ratings  only  on  the 
SACHA-generated  version  of  Rule  Application,  de¬ 
fined  as  the  ability  to  efficiently  apply  transformational 
rules  inferred  from  the  complete  portions  of  the  stimu¬ 
lus  array  to  the  incomplete  portion  of  the  array.  The 
SME-generated  version  of  Rule  Application,  which  was 
defined  as  the  ability  to  apply  learned  rules  to  the  real  world 
work  situation,  was  not  included  in  the  linkage  survey. 

Respondent  Background  Questionnaire 

Project  staff  created  the  7-item  background  ques¬ 
tionnaire  to  be  completed  by  the  survey  respondents. 
Items  measured  the  respondent’s  highest  educational 
degree,  area  of  study,  experience  in  data  collection, 
experience  in  test  construction,  familiarity  with  AT- 
SAT,  role  in  developing  AT-SAT  predictors  and/or 
criterion  measures,  and  familiarity  with  ATCS  job.  One 
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purpose  of  the  items  was  to  serve  as  a  check  in  making 
sure  the  individuals  were  qualified  raters.  Additionally, 
these  items  could  serve  to  identify  subgroups  of  raters 
based  upon  such  things  as  testing  experience,  educa¬ 
tional  background,  and/or  educational  degree.  In  the 
event  that  rater  reliability  was  low,  attempts  could  be 
made  to  determine  whether  the  lack  of  rater  agreement 
was  due  to  any  one  of  these  subgrouping  variables. 

Descriptions  of  AT-SAT  Measures 

The  AT-SAT  test  battery  used  in  the  concurrent 
validity  study  contained  the  following  12  predictor 
measures: 

•  Dials 

•  Sound 

•  Letter  Factory 

■  Applied  Math 

■  Scanning 

•  Angles 

•  Analogies 

•  Memory 

•  Air  Traffic  Scenarios 

•  Experience  Questionnaire 

•  Time  Wall/Pattern  Recognition 

•  Planes 

An  important  element  of  the  linkage  survey  consisted 
of  operational  descriptions  of  each  of  the  AT-SAT 
predictor  tests.  These  descriptions  were  meant  not  to 
replace  the  respondent’s  familiarity  and  experience  with 
each  of  the  measures  but  to  highlight  the  most  basic 
features  of  each  measure  for  each  respondent.  While  one 
of  the  criteria  for  inclusion  as  a  survey  respondent  was 
a  familiarity  with  each  of  the  measures  (typically  gained 
through  helping  to  create  and/or  taking  the  test),  it  was 
felt  that  a  general  description  of  the  features  of  each 
measure  would  facilitate  a  more  complete  recall  of  the 
test  characteristics.  Respondents  were  instructed  to  read 
each  test  description  before  rating  the  degree  to  which 
that  test  measures  each  of  the  WRs.  Figure  2.1  is  an 
example  of  a  description  for  one  of  the  AT-SAT  mea¬ 
sures  and  its  accompanying  rating  scale. 

The  only  predictor  measure  for  which  no  operational 
description  was  provided  was  the  Experience  Question¬ 
naire  (EQ).  This  measure  was  a  biodata  inventory 
comprised  of  14  subscales,  with  individual  subscales 
containing  anywhere  from  9  to  1 5  items.  In  the  place  of 
test  descriptions,  respondents  making  linkage  ratings 
on  the  EQ  received  the  actual  items  from  the  individual 


scales  (but  did  not  receive  the  construct  labels  for  these 
scales).  Respondents  were  to  use  the  items  comprising 
each  scale  to  determine  the  construct  being  measured  by 
that  particular  scale  and  then  make  their  ratings  as  to  the 
degree  to  which  the  scale  successfully  measured  each  WR 

Definitions  of  WRs 

The  survey  contained  an  attachment  listing  the  WRs 
and  their  accompanying  definitions  from  SACHA’s 
revised  consolidated  WR  list  (except  for  the  SME- 
generated  WR  of  Rule  Application).  It  was  felt  that,  in 
order  for  respondents  to  make  the  most  informed  link¬ 
age  rating  between  a  test  and  a  WR,  they  should  not  only 
have  a  clear  understanding  of  the  properties  of  the  test, 
but  also  possess  a  firm  grasp  of  the  WR.  Survey  respon¬ 
dents  were  instructed  to  read  through  the  attachment  of 
WRs  and  their  respective  definitions  before  making  any 
linkage  ratings  and  to  refer  back  to  these  definitions 
throughout  the  rating  process  (Table  2.8). 

Survey  Respondents 

To  qualify  as  raters,  individuals  had  to  be  familiar 
with  the  measures  comprising  the  AT-SAT  battery,  and 
they  had  to  have  an  understanding  of  each  of  the  WRs 
being  linked  to  the  various  measures.  Potential  respon¬ 
dents  were  contacted  by  phone  or  E-mail  and  informed 
of  the  nature  of  the  rating  task.  A  pool  of  25  potential 
respondents  was  identified.  The  individuals  in  this  pool 
came  primarily  from  the  organizations  contracted  to 
perform  the  AT-SAT  validation  effort  but  also  included 
FAA  personnel  directly  involved  with  AT-SAT. 

Survey  Methodology 

Those  who  had  agreed  to  participate  in  the  linkage 
process  received  the  packet  of  rating  materials  via  regu¬ 
lar  mail.  Each  packet  contained  the  following  items: 

( 1 )  An  introduction,  which  outlined  the  importance  of 
linking  the  AT-SAT  predictor  tests  to  the  WRs  identi¬ 
fied  in  the  SACFIA  job  analysis.  It  included  the  names 
and  phone  numbers  of  project  staff  who  could  be 
contacted  if  respondents  had  questions  concerning  the 
rating  process. 

(2)  The  7-item  background  questionnaire. 

(3)  The  attachment  containing  the  list  of  WRs  and 
their  definitions. 


(4)  Rating  scales  for  each  of  the  AT-SAT  tests.  Each 
rating  scale  contained  the  operational  description  of  the 
measure  (or  for  the  EQ,  those  items  comprising  an  EQ 
sub-scale),  the  Likert  scale  response  options,  and  the 
WRs  to  be  rated  (Figure  2.2.). 

In  view  of  the  inordinate  amount  of  time  it  would 
take  a  respondent  to  rate  all  the  WRs  on  each  of  the  12 
tests,  tests  were  divided  in  half,  with  one  group  of 
respondents  rating  six  tests  and  the  other  group  rating 
the  other  six  tests.  The  25  potential  raters  who  had  been 
identified  were  split  into  two  groups.  Thirteen  respon¬ 
dents  were  responsible  for  linkage  ratings  for  the  Angles, 
Analogies,  Memory,  AT  Scenarios,  Planes,  and  Experi¬ 
ences  Questionnaire;  the  remaining  12  were  to  make 
linkage  ratings  for  the  Dials,  Sound,  Letter  Factory, 
Applied  Math,  Scanning,  and  Time  Wall.  The  subset  of 
tests  sent  to  the  1 3  respondents  was  labeled  Version  1 , 
and  the  second  subset  of  tests  sent  to  the  remaining  12 
respondents  was  labeled  Version  2. 

Results  of  the  Survey 

The  surveys  were  returned,  and  the  data  were  ana¬ 
lyzed  by  project  staff.  Twenty-four  respondents  com¬ 
pleted  the  background  questionnaire,  as  well  as  all  or 
portions  of  the  six  tests  they  were  to  link  with  the 
respective  WRs.  Nineteen  of  the  24  respondents  classi¬ 
fied  themselves  as  Industrial/Organizational  Psycholo¬ 
gists;  all  but  one  of  the  24  had  obtained  at  least  a  master’s 
degree.  In  general,  results  of  the  questionnaire  indicated 
that  the  raters  were  experienced  in  test  construction  and 
administration  and  were  familiar  with  the  AT-SAT  test 
battery. 

All  12  respondents  who  were  asked  to  rate  Version  1 
of  the  linkage  survey  completed  and  returned  their 
ratings.  Two  of  these  individuals  volunteered  to  com¬ 
plete  the  linkage  ratings  of  the  Version  2  tests  and 
followed  through  by  completing  these  ratings  as  well. 
Completed  ratings  were  returned  by  11  of  the  13 
respondents  who  were  tasked  with  making  linkage 
ratings  for  Version  2  tests,  with  one  rater  choosing  not 
to  rate  either  the  Memory  or  the  Planes  tests.  Hence,  12 
complete  sets  of  linkage  ratings  were  obtained  for  the 
Version  1  tests,  and  14  complete  sets  of  linkage  ratings 
were  obtained  for  all  but  two  of  the  Version  2  tests 
(Table  2.9). 


Scale  Reliability 

Reliability  indices  were  computed  for  each  rating 
scale.  Scale  reliabilities  ranged  from  .86  to  .96.  Hence, 
the  intraclass  correlations  (Shrout  &  Fleiss,  1979)  for 
each  of  the  rating  scales  revealed  a  high  level  of  agree¬ 
ment  between  the  respondents  as  to  which  WRs  were 
being  successfully  measured  by  the  respective  tests. 
These  reliability  coefficients  are  listed  in  Table  2.9.  In 
view  of  the  high  level  of  agreement,  it  appeared  that  such 
factors  as  the  rater’s  highest  educational  degree,  educational 
background,  and  familiarity  with  the  ATCS  job  did  not 
influence  the  level  of  agreement  among  the  raters. 

Angles 

The  Angles  test  measures  the  participant’s  ability  to 
recognize  angles.  This  test  contains  30  multiple-choice 
questions  and  allows  participants  up  to  8  minutes  to 
complete  them.  The  score  is  based  on  the  number  of 
correct  answers  (with  no  penalty  for  wrong  or  unan¬ 
swered  questions).  There  are  two  types  of  questions  on 
the  test.  The  first  presents  a  picture  of  an  angle  and  the 
participant  chooses  the  correct  answer  of  the  angle  (in 
degrees)  from  among  four  response  options.  The  second 
presents  a  measure  in  degrees  and  the  participant  chooses 
the  angle  (among  four  response  options)  that  represents 
that  measure.  For  each  worker  requirement  listed  be¬ 
low,  enter  the  rating  best  describing  the  extent  to  which 
this  test  and/or  its  subtests  measure  that  particular 
worker  requirement. 

5=  This  test  measures  this  worker  requirement  to  a 
very  great  extent 

4=  This  test  measures  this  worker  requirement  to  a 
considerable  extent 

3=  This  test  measures  this  worker  requirement  to  a 
moderate  extent 

2=  This  test  measures  this  worker  requirement  to  a 
limited  extent 

1=  This  test  measures  this  worker  requirement  to  a 
slight  extent 

0=  This  test  does  not  measure  this  worker  require¬ 
ment 

Linkage  Results 

Mean  linkage  scores  between  tests  and  WRs  were 
computed,  representing  a  summary  of  the  extent  to 
which  the  raters  felt  each  test  measured  each  WR.  It  was 
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decided  that  linkage  means  greater  than  or  equal  to  3 
suggested  that  the  raters  felt  a  test  was  able  to  measure 
the  WR  to  at  least  a  moderate  extent.  Therefore,  a 
criterion  cutoff  mean  >  3  was  established  to  determine 
if  a  test  was  indeed  able  to  successfully  measure  a 
particular  WR.  T able  2.10  presents  the  following  infor¬ 
mation:  (a)  a  full  list  of  the  WRs  (rank  ordered  by  their 
importance  for  doing  the  job),  (b)  those  tests  rated  as 
measuring  the  WR  to  at  least  a  moderate  extent  (based 
upon  the  mean  >  3  cutoff),  and  (c)  the  mean  linkage 
rating  corresponding  to  that  test/WR  pair. 

The  rank-ordered  listing  of  WRs  used  in  Table  2.10 
was  derived  from  the  SACHA  job  analysis  and  consisted 
of  the  ARTCC  controllers’  ratings  of  the  extent  to 
which  the  WR  was  seen  as  being  important  for  doing  the 
job.  Hence,  on  the  basis  of  ARTCC  controllers’  rating 
from  SACHA,  prioritization  was  seen  as  the  most 
important  WR  for  doing  the  job.  AT-SAT  project  staff 
chose  to  use  the  ARTCC  rank-ordered  listing  of  WRs 
on  their  importance  for  doing  the  job  for  three  reasons. 
First,  the  ARTCC  list  was  chosen  over  the  list  generated 
by  the  ratings  from  all  ATCSs  because  the  latter  in¬ 
cluded  the  ratings  of  controllers  from  the  Flight  Service 
option.  SACFIA  had  clearly  stated  that  the  job  analysis 
findings  for  the  Flight  Service  option  were  different 
enough  from  the  ARTCC  and  Terminal  options  to 
require  its  own  predictor  development.  Second,  the 
ARTCC  list  was  selected  over  the  Terminal  option  list 
because  the  AT-SAT  validation  effort  was  using  con¬ 
trollers  from  the  ARTCC  option.  Finally,  the  ARTCC 
list  for  doing  the  job  was  chosen  over  the  list  for  learning 
the  job  because  the  focus  ofAT-SAT  was  on  developing 
a  selection  battery  that  would  predict  performance  in 
the  job — not  necessarily  in  training. 

It  should  be  mentioned  that  no  data  on  importance 
for  learning  or  doing  the  job  existed  for  five  of  the 
SACHA-generated  information  processing  WRs  (Con¬ 
firmation,  Encoding,  Rule  Inference,  Rule  Application, 
and  Learning).  This  was  because  the  SACHA  project 
staff  believed  that  controllers  could  not  adequately 
comprehend  these  WRs  well  enough  to  rate  them. 
Because  of  this  lack  of  importance  data,  project  staff  placed 
these  WRs  at  the  end  of  the  list  of  WRs  (see  Table  2.10.). 

Based  upon  these  criteria  for  inclusion,  it  was  found 
that  14  of  the  15  most  important  WRs  (as  rated  by 
ARTCC  controllers  in  the  SACHA  job  analysis)  were 
successfully  measured  by  one  or  more  of  the  tests  of  the 
AT-SAT  battery.  Similarly,  the  mean  linkage  ratings 
suggest  that  the  vast  majority  of  the  more  important 
WRs  were  successfully  measured  by  multiple  tests. 


The  linkage  survey  results  indicated  that  all  impor¬ 
tant  WRs  were  not  successfully  measured  by  the  AT- 
SAT  battery.  Four  WRs  (Oral  Communication,  Problem 
Solving,  Long-Term  Memory,  and  Visualization)  from 
the  top  third  of  SACFIA’s  rank-ordered  list  did  not  have 
linkage  means  high  enough  to  suggest  that  they  were 
being  measured  to  at  least  a  moderate  extent.  None  of 
the  AT-SAT  tests  were  specifically  designed  to  measure 
oral  communication  and,  as  a  result,  linkage  means 
between  this  WR  and  the  tests  were  found  to  be  at  or 
near  zero.  Problem  Solving  had  mean  linkage  ratings 
that  approached  our  criterion  for  inclusion  for  the 
Applied  Math  and  the  Letter  Factory  tests.  Similarly,  the 
mean  linkage  ratings  between  the  Memory  test  and 
Long-Term  Memory,  and  between  the  Letter  Factory 
test  and  Visualization  also  approached  but  failed  to 
meet  the  mean  criterion  score  of  3. 

Quality  of  Individual  Tests  in  the  AT-SAT  Battery 

Results  of  the  linkage  survey  were  also  summarized  to 
enable  project  staff  to  gain  insight  into  how  well  indi¬ 
vidual  tests  were  measuring  the  most  important  WRs. 
Based  upon  the  criterion  of  mean  linkage  score  >  3  for 
demonstrating  that  a  test  successfully  measures  a  par¬ 
ticular  WR,  project  staff  determined  the  number  of 
WRs  successfully  measured  by  each  test.  This  score 
provided  some  indication  of  the  utility  of  each  test. 
Project  staff  also  computed  two  additional  scores  to 
indicate  the  utility  of  each  measure.  Some  WRs  were 
rated  as  being  successfully  measured  by  many  tests,  and 
other  WRs  were  measured  by  only  one  or  two  tests.  Two 
other  indicators  of  the  utility  of  a  measure  were  devel¬ 
oped:  (a)  the  number  of  WRs  a  test  measured  that  are 
only  measured  by  one  (or  fewer)  other  test(s),  and  (b)  the 
number  of  WRs  that  are  not  measured  by  any  other  test. 
Scores  based  upon  rhese  criteria  were  computed  for  each 
measure  and  are  listed  in  Table  2.1 1. 

In  addition  to  the  indicators  of  each  test’s  utility,  it 
was  felt  that  indicators  of  each  test’s  utility  and  quality 
in  measuring  WRs  could  also  be  computed.  To  provide 
some  indication  of  each  test’s  quality,  project  staff  again 
utilized  SACHA  findings —  the  ARTCC  controller 
ratings  of  the  importance  of  each  WR  for  doing  the  job. 
Each  WR’s  mean  importance  rating  (from  SACHA) 
was  multiplied  by  those  WR/test  linkage  ratings  meet¬ 
ing  criteria.  The  product  of  these  two  scores  (mean  WR 
importance  for  doing  the  job  x  mean  linkage  rating  of 
WR  for  a  test)  factored  in  not  only  how  well  the  test  was 
capturing  the  WR  but  the  importance  of  that  WR  as 
well.  The  mean  and  sum  of  these  products  were  com- 
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puted  for  each  (Table  2. 1 1  .)■  The  mean  of  the  products 
can  be  viewed  as  an  indicator  of  the  average  quality  of  a 
measure  factoring  in  both  how  well  the  test  was  measur¬ 
ing  the  WR  and  the  importance  of  the  WR.  The  sum  of 
the  products  provides  some  indication  of  the  overall 
utility  of  the  measure  in  that  the  more  WRs  a  test 
captures,  the  better  it  captures  those  WRs,  and  the  more 
important  these  WRs  are  for  the  doing  the  job,  the 
higher  a  test’s  total  score  on  this  factor. 

Given  that  no  data  were  collected  in  SACHA  for  five 
of  the  WRs  (Confirmation,  Encoding,  Rule  Inference, 
Rule  Application,  and  Learning)  on  their  importance 
for  doing  the  job,  the  mean  importance  score  across  all 
the  WRs  was  imputed  for  these  five  WRs.  This  was  done 
so  that  some  indication  of  a  test’s  ability  to  measure 
these  WRs  could  be  computed  and  factored  into  its 
overall  quality  and  utility  scores  (Table  2.1 1.). 

Results  suggest  that  some  tests  -  Letter  Factory,  AT 
Scenarios,  and  to  a  lesser  degree  the  Time  Wall  and 
Analogies  tests  -  measured  numerous  WRs,  while  the 
remaining  tests  measured  from  one  to  three  WRs.  Some 


tests,  such  as  Applied  Math  and  Analogies,  measured 
multiple  WRs  that  were  not  measured  by  other  tests, 
while  other  tests  (Letter  Factory,  Air  Traffic  Scenarios, 
and  Time  Wall)  measured  many  WRs  but  none  uniquely. 
It  should  be  mentioned  that  one  of  the  reasons  the  Letter 
Factory,  Air  Traffic  Scenarios,  and  Time  Wall  did  not 
uniquely  capture  any  WRs  was  that  there  was  so  much 
overlap  in  the  WRs  successfully  measured  by  these  three 
tests — especially  between  the  Letter  Factory  and  the  Air 
Traffic  Scenarios. 

Conclusion 

Based  upon  the  results  of  the  linkage  survey,  every 
test  within  the  AT-SAT  battery  appeared  to  be  success¬ 
fully  measuring  at  least  one  WR,  and  many  of  the  tests 
were  rated  as  measuring  multiple  WRs.  While  not  every 
WR  was  thought  to  be  successfully  measured  by  the  AT- 
SAT  battery,  the  vast  majority  of  the  WRs  considered 
most  important  for  doing  the  job  was  successfully 
measured  by  one  or  more  predictors  from  the  battery. 
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CHAPTER  3.1 


Predictor  Development  Background 

Douglas  Quartetti,  HumRRO 
William  Kieckhaefer,  RGI,  Inc. 

Janis  Houston,  PDRI,  Inc 

The  final  test  in  the  OPM  Battery,  the  OKT,  con- 


Following  the  air  traffic  controller  strike  and  the 
subsequent  firing  of  a  significant  portion  of that  workforce 
in  1 98 1 ,  the  Federal  Aviation  Administration  was  forced 
to  hire  en  masse  to  ensure  safety  of  the  airways.  Cooper, 
Blair,  and  Schemmer  (1994)  reported  on  the  selection 
procedures  used  after  the  strike.  Their  work  is  summa¬ 
rized  below. 

SELECTION  PROCEDURES  PRIOR  TO 
AT-SAT 

The  OPM  Battery 

In  October  1981,  the  FAA  introduced  a  two-stage 
process  for  selecting  Air  Traffic  Control  Specialists 
(AT CSs) .  The  first  stage  was  a  paper-and-pencil  aptitude 
test  battery  administered  by  the  Office  of  Personnel 
Management  (OPM),  known  as  the  OPM  Battery.  This 
battery  consisted  of  three  tests:  the  Multiplex  Controller 
Aptitude  Test  (MCAT),  the  Abstract  Reasoning  Test 
(ABSR),  and  the  Occupational  Knowledge  Test  (OKT). 
The  second  stage  was  called  the  Academy  Screen. 

The  first  test  of  the  OPM  Battery,  the  MCAT,  simu¬ 
lated  aspects  of  air  traffic  control.  Applicants  were  re¬ 
quired  to  solve  time,  distance,  and  speed  problems,  plus 
interpret  tabular  and  graphical  information  to  identify 
potential  conflicts  between  aircraft.  Extensive  research  at 
the  Civil  Aeromedical  Institute  (CAMI)  indicated  that 
the  MCAT  scores  were  significantly  correlated  with 
performance  during  the  Academy  Screen  and  later  field 
status  (Manning,  Della  Rocco,  &  Bryant,  1989;  Rock, 
Dailey,  Ozur,  Boone,  &  Pickerel,  1978). 

The  second  test  of  the  OPM  Battery,  the  ABSR,  was 
developed  by  the  U.S.  Civil  Service  Commission  to 
examine  the  abstract  relationships  between  symbols  and 
letters.  Research  indicated  a  relationship  between  scores 
on  this  test  and  the  Academy  Screen  training  perfor¬ 
mance  (Boone,  1979;  Rocket  al.,  1978). 


tained  questions  on  air  traffic  phraseology  and  proce¬ 
dures.  It  was  designed  to  provide  credit  for  prior  ATCS 
experience.  It  has  been  reported  that  OKT  scores  corre¬ 
lated  with  many  of  the  indices  of  training  success  (Boone, 
1979;  Buckley,  O’Connor,  &  Beebe,  1970;  Manning  et 
al.,  1989;  Mies,  Coleman,  &  Domenech,  1977). 

The  scores  on  the  MCAT  and  the  ABSR  were  com¬ 
bined  with  weights  of  .80  and  .20  applied,  respectively. 
These  scores  were  then  transmuted  to  have  a  mean  of  70 
and  maximum  of  100.  The  passing  score  varied  with 
education  and  prior  experience.  Applicants  who  received 
passing  scores  on  the  first  two  predictors  could  receive  up 
to  15  additional  points  from  the  OKT. 

The  second  stage  in  the  hiring  process  was  the  Acad¬ 
emy  Screen.  Applicants  who  passed  the  OPM  Battery 
were  sent  to  the  FAA  Academy  for  a  9-week  screen, 
which  involved  both  selection  and  training  (Manning, 
1991a).  Students  spent  the  first  5  weeks  learning  aviation 
and  air  traffic  control  concepts  and  the  final  4  weeks 
being  tested  on  their  ability  to  apply  ATC  principles  in 
non-radar  simulation  problems.  Applicants  could  still  be 
denied  positions  after  the  9  weeks  on  the  basis  of  their 
scores  during  this  phase.  The  reported  failure  rate  was  40 
percent  (Cooper  et  al.,  1994). 

This  hiring  process  received  much  criticism,  despite 
its  reported  effectiveness  and  links  to  job  performance. 
The  criticisms  revolved  around  the  time  (9  weeks  for  the 
Academy  screen)  and  cost  of  such  a  screening  device 
($10,000  per  applicant).  In  addition  to  the  FAA  invest¬ 
ment,  applicants  made  a  substantial  investment,  and  the 
possibility  remained  that  after  the  9  weeks  an  applicant 
could  be  denied  a  position.  Finally,  there  was  concern 
that  the  combination  of  screening  and  training  reduced 
training  effectiveness  and  made  it  impossible  to  tailor 
training  needs  to  individual  students. 

As  a  result  of  these  criticisms,  the  FAA  separated 
selection  and  training,  with  the  idea  that  the  training 
atmosphere  of  the  Academy  Screen  would  be  more 
supportive  and  oriented  toward  development  of  ATCSs 
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once  it  was  separated  from  selection.  This  necessitated 
developing  a  new  selection  device  to  replace  the  Acad¬ 
emy  Screen. 

The  Pre-Training  Screen 

The  FAA  introduced  the  Pre-Training  Screen  (PTS) 
in  June  1992  to  replace  the  second  stage  of  the  hiring 
process,  the  Academy  Screen.  The  PTS  was  developed 
from  a  cognitive  psychology  perspective  by  Aerospace 
Sciences,  Inc.  (ASI)  in  1991.  It  was  computer  adminis¬ 
tered  and  consisted  of  two  parts:  the  Complex  Cogn  itive 
Battery  and  the  Air  T raffic  Scenario  Test.  For  complete 
descriptions  of  the  components  of  the  PTS  and  the 
critical  aptitudes  covered  by  these  tests,  the  reader  is 
referred  to  ASI  (1991). 

The  first  part  of  the  PTS,  the  Complex  Cognitive 
Battery,  included  five  test  components:  Static  Vector/ 
Continuous  Memory,  Time  Wall/Pattern  Recognition, 
Visual  Search,  Stroop  Color-Word  Test,  and  Letter 
Rotation  Test.  According  to  ASI  (1991),  the  Static 
Vector/Continuous  Memory  Test  was  a  multimeasure 
test  designed  to  assess  the  critical  aptitudes  of  spatial 
relations,  working  memory,  verbal/numerical  coding, 
attention  switching,  and  visualization.  The  Time  Wall/ 
Pattern  Recognition  test  was  designed  to  assess  filtering, 
movement  detection,  prioritizing,  short-term  memory, 
image/pattern  recognition,  and  spatial  scanning.  The 
Visual  Search  test  measured  short-term  memory  and 
perceptual  speed.  The  Stroop  Color-Word  Test  assessed 
the  critical  aptitudes  of  decoding,  filtering,  and  short¬ 
term  memory.  Finally,  the  Letter  Rotation  Test  assessed 
the  critical  aptitudes  of  decoding,  image/pattern  recog¬ 
nition,  and  visualization.  It  should  be  noted  that  each 
test  in  this  battery  could  yield  multiple  scores. 

The  second  part  of  the  PTS,  the  Air  Traffic  Scenario 
Test,  was  a  low-fidelity  work  sample  test  (Broach  & 
Brecht-Clark,  1 994) .  Applicants  were  given  a  synthetic, 
simplified  air  space  to  control.  This  test  was  designed  to 
assess  nearly  all  of  the  critical  aptitudes  of  the  ATCS  job. 

Two  attempts  were  made  to  validate  the  PTS.  The 
first  (ASI,  1991)  correlated  PTS  performance  with 
training  criteria  (the  Academy  Screen  Comprehensive 
Test  score).  Based  on  correlation  analyses,  the  full  set  of 
test  scores  was  reduced  to  ten  (Safety  and  Delay  scores 
from  Air  Traffic  Scenario;  Percent  Correct  and  Mean 
Correct  Reaction  Time  from  Static  Vector;  Percent 
Correct  and  Mean  Correct  Reaction  Time  from  Con¬ 
tinuous  Memory;  Mean  Absolute  Time  Error  from 


Time  Wall;  Mean  Correct  Reaction  Time  from  Pattern 
Recognition;  Stroop  Mean  Reaction  Time  for  Conflict 
Stimuli  from  the  Stroop  Color-Word  Test;  and  Visual 
Search  Mean  Correct  Reaction  Time  from  Visual 
Search).  These  scores  were  retained  based  on  their 
single-order  correlation  with  the  criterion,  their 
intercorrelations  with  other  predictor  scores,  and  the 
multiprocessing  nature  of  the  paired  test  scores  (e.g..  Air 
Traffic  Safety  and  Delay). 

Multiple  regression  analyses  showed  that  the  Safety 
score  from  Air  Traffic  Scenario  and  the  Percent  Correct 
and  Correct  Reaction  Time  scores  from  the  Static 
Vector  test  had  significant  power  in  predicting  the 
Academy  Screen  Comprehensive  Test  score.  The  betas 
for  the  remaining  subtest  scores  were  not  significant  in 
the  context  of  the  other  tests.  ASI  (1991)  reported  the 
regression  model  shown  in  Tables  3.1.1  and  3.1.2. 

The  Pre-Training  Screen  was  intended  to  be  used  as 
a  secondary  screening  procedure.  Incremental  validity 
was  estimated  for  the  OPM  battery  score,  and  for  the 
OPM  and  PTS  scores  where  the  OPM  score  was  entered 
in  step  1  and  the  PTS  scores  were  entered  in  a  block  at 
step  2.  The  OPM  score  alone  produced  an  R  =  .226,  R 
square  =  .05.  The  model  using  the  OPM  and  PTS  scores 
produced  a  multiple  correlation  of  R  =  .505,  R  square  = 
.26.  The  difference  of  variance  accounted  for  by  the 
addition  of  the  PTS  (.26  vs.  .05)  was  significant  (F_= 
24.18,  p  <.01).  This  indicated  that  the  Pre-Training 
Screen  added  significantly  to  the  prediction  of  the 
Academy  Screen  Comprehensive  Test  score,  over  and 
above  the  OPM  battery  alone. 

The  second  validation  attempt  (Weltin,  Broach, 
Goldbach,  &  O’Donnell,  1 992)  was  a  concurrent  crite¬ 
rion-related  study  using  a  composite  measure  of  on-the- 
job  training  performance.  Scores  obtained  from  the  Air 
Traffic  Scenario  Test,  Static  Vector,  Continuous 
Memory,  Stroop  Color-Word  Test,  and  Letter  Rota¬ 
tion  Test  correlated  significantly  with  the  criterion. 
Using  two  weighting  schemes,  the  regression-based 
weighting  scheme  yielded  a  correlation  of  .21,  whereas 
the  unit  weighting  yielded  a  correlation  of  .18. 

Use  of  the  PTS  as  a  screening  device  was  discontin¬ 
ued  in  February  1 994.  The  defensibility  of  the  PTS  was 
questioned  since  it  was  validated  only  against  training 
performance  criteria.  The  perception  was  that  the  test 
security  of  the  OPM  test,  in  use  since  1981  without 
revision,  had  been  compromised.  Further,  several  coach¬ 
ing  schools  provided  guarantees  to  students  that  they 
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would  pass  the  OPM  battery.  For  a  more  complete 
discussion  of  prior  programs  used  to  screen  ATCS 
candidates  before  AT-SAT,  see  Chapter  6  of  this  report. 

Separation  and  Control  Hiring  Assessment 
(SACHA) 

In  September  1991,  the  FAA  awarded  a  contract  to 
University  Research  Corporation  for  the  development 
and  validation  of  a  new  test  battery  for  selection  of 
AT  CSs.  (The  outcomes  of  the  SACFIA  job  analysis  were 
covered  in  more  detail  in  Chapter  2.)  By  1996,  a 
comprehensive  job  analysis  was  completed  on  four 
ATCS  options,  and  the  construction  of  possible  predic¬ 
tor  tests  had  begun.  The  FAA  terminated  the  SACHA 
contract  late  in  1996. 

A  meta-analytic  study  of  much  of  the  previous  vali¬ 
dation  research  on  the  ATCS  job  was  performed  as  part 
of  SACFIA  (Schemmer  et  al. ,  1 996) .  This  study  reported 
on  predictors  ranging  from  traditional  cognitive  ability 
tests,  and  personal  characteristics  instruments,  to  air  traffic 
control  simulations  and  psychomotor  ability  measures. 
The  validity  studies  are  summarized  in  Table  3.1.3. 

As  reported  by  Schemmer  et  al.  (1996),  for  most  of 
the  predictor  measure  categories,  the  validity  coeffi¬ 
cients  exhibited  substantially  greater  variability  than 
would  be  expected  under  the  simple  explanation  of 
sampling  error.  This  suggests  that,  in  general,  some 
specific  predictor  measures  are  relatively  more  predic¬ 
tive  of  job  performance  than  others.  For  example, 
simulations  and  math  tests  have  historically  been  good 
predictors  of  controller  job  performance. 

On  the  basis  of  the  SACHA  job  analysis,  Schemmer 
et  al.  (1996)  proposed  an  overall  model  of  the  ATCS 
worker  requirements  that  included  a  Cognitive  Model 
and  a  Temperament/Interpersonal  Model.  The  Cogni¬ 
tive  Model  contained  two  higher-order  constructs,  g 
and  Processing  Operations.  Table  3.1.4  displays  the 
construct  categories,  worker  requirements  under  the 
higher  order  construct  of  g,  and  the  tests  purported  to 
measure  the  worker  requirements.  Schemmer  et  al. 
recommended  at  least  one  test  per  worker  requirement. 
As  Table  3. 1 .4  shows,  there  were  some  worker  require¬ 
ments  for  which  the  project  still  had  not  developed  tests. 
For  example,  their  predictor  battery  did  not  account  for 
any  of  the  requirements  under  the  rubric  of  Communi¬ 
cation.  Additionally,  much  of  the  Applied  Reasoning 
construct  remained  untested,  and  Numeric  Ability 
(Multiplication/Division),  Scanning,  and  Movement 
Detection  were  not  addressed. 


Table  3.1.5  displays  the  construct  categories,  worker 
requirements  under  the  higher  order  construct  of  Pro¬ 
cessing  Operations,  and  the  tests  that  Schemmer  et  al. 
hypothesized  would  assess  the  worker  requirements. 
Table  3.1.5  reveals  that,  for  the  construct  labeled 
Metacognitive,  no  tests  had  been  recommended.  In 
addition,  Schemmer  et  al.  did  not  account  for  Sustained 
Attention,  Timesharing,  Scanning,  or  Movement  De¬ 
tection  worker  requirements. 

Finally,  a  Temperament/Interpersonal  Model  was 
proposed  to  provide  coverage  of  the  worker  require¬ 
ments  that  did  not  fit  into  the  Cognitive  Model  (Table 
3.1.6.). 

As  noted  earlier,  due  to  a  compromised  OPM  battery 
and  the  elimination  of  use  of  PTS,  the  FAA  decided  to 
support  the  development  and  validation  of  a  new  test 
battery  against  job  performance  criteria.  With  this  deci¬ 
sion,  a  contract  was  awarded  to  Caliber  Associates,  and 
support  for  the  AT-SAT  project  was  initiated. 

AIR  TRAFFIC  SELECTION  AND 
TRAINING  (AT-SAT)  PROJECT 

One  of  the  challenges  facing  the  AT-SAT  research 
team  was  to  decide  what  SACHA-generated  materials 
would  be  adequate  for  the  new  battery  and  how  many 
new  tests  needed  to  be  developed.  This  section  describes 
the  procedures  undertaken  to  review  existing  SACHA 
materials  and  documents  the  evaluation  of  and  com¬ 
ments  on  the  battery’s  coverage  of  the  predictor  space. 
Recommendations  for  the  AT-SAT  project  were  made, 
based  on  the  review  process. 

Test  by  Test  Evaluation 

A  panel  of  nine  individuals  was  asked  to  review  each 
test  currently  available  on  computer  for  possible  inclu¬ 
sion  in  the  air  traffic  control  predictor  test  battery. 
Evaluation  sheets  were  provided  for  each  test,  request¬ 
ing  information  about  the  following  criteria: 

(1)  Does  the  test  measure  the  worker  requirement(s)  it 
purports  to  measure? 

(2)  Is  it  a  tried-and-true  method  of  assessing  the  worker 
requirement(s)? 

(3)  Does  the  scoring  process  support  the  measurement 
of  the  worker  requirement(s)? 

(4)  Is  the  time  allocation/emphasis  appropriate? 

(5)  Is  the  reading  level  consistent  with  job  require¬ 
ments? 
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(6)  Does  the  test  have  potential  adverse  impact? 

(7)  Is  the  test  construction  ready  for  validation  admin¬ 
istration? 

Short  descriptions  of  each  test  were  provided  for  this 
evaluation,  along  with  test  information  such  as  number 
of  items,  and  scoring  procedures.  The  worker  require¬ 
ment  definitions  used  throughout  this  evaluation  pro¬ 
cess  were  those  listed  for  the  Revised  Consolidated 
Worker  Requirements  on  pages  115-119  of  the  SA  CHA 
Final  Job  Analysis  Report  Qanuary  1995).  Sixteen  tests 
were  independently  reviewed  by  the  panel  members. 
The  results  of  the  independent  reviews  were  discussed  at 
a  3-day  meeting.  An  additional  four  tests  (Letter  Factory 
and  the  three  PTS  tests)  were  reviewed  in  a  similar 
fashion  during  the  meeting.  The  20  tests  reviewed  were: 


Sound 

Scan 

Angles 

Map 

Dial  Reading 
Headings 
Projection 
Memory  1  and  2 
Direction  and  Distance 
Planes 


Stix 

Time 

Syllogism 

Analogy 

Classification 

Personal  Experiences  and  Attitude 
Questionnaire  (PEAQ) 

Letter  Factory 

Air  Traffic  Scenario  (from  PTS) 
Time  Wall/Pattern  Recognition 
(from  PTS)  Static  Vector/Con¬ 
tinuous  Memoty  (from  PTS) 


The  project  staff  met  with  the  panel  members  on  5- 
7  November,  1996  to  discuss  the  predictor  battery.  For 
each  test,  independent  ratings  on  each  evaluation  crite¬ 
rion  were  collected,  and  the  relative  merits  and  prob¬ 
lems  of  including  that  test  in  the  predictor  bartery  were 
discussed.  The  comments  were  summarized  and  re¬ 
corded. 

After  the  group  discussion,  panel  members  were 
asked  to  provide  independent  evaluations  on  whether  or 
not  each  test  should  be  included  in  the  predictor  battery. 
For  each  test,  panel  members  indicated  “Yes"  for  inclu¬ 
sion,  “No”  for  exclusion,  and  “Maybe"  for  possible 
inclusion.  The  Yes-No-Maybe  ratings  were  tallied  and 
summarized. 


Selection  of  a  Subset  of  Tests 

The  next  step  involved  selecting  a  subset  of  the  20 
tests  for  inclusion  in  the  predictor  battery.  Consider¬ 
ations  included  both  the  Yes-No-Maybe  ratings  (based 
on  multiple,  test-specific  criteria),  and  how  well  differ¬ 


ent  subsets  of  tests  covered  the  predictor  domain.  The 
list  ofworker  requirements,  rank  ordered  by  incumbent 
importance  ratings,  as  described  in  Chapter  2,  was  used 
to  help  determine  whether  different  tests  or  subsets  of 
tests  covered  the  critical  job  requirements.  These  inves¬ 
tigations  and  recommendations  are  summarized  below. 

Nine  tests  that  received  a  preponderance  of  Yes 
ratings  were  measuring  critically  important  job  require¬ 
ments  and  appeared  to  be  relatively  non-overlapping. 
These  were  Scan,  Letter  Factory,  Sound,  Dial  Reading, 
PEAQ,  Analogy,  Air  Traffic  Scenario  (ATS),  Time 
Wall/Pattern  Recognition  (TW),  and  Static  Vector/ 
Continuous  Memory  (SV).  These  nine  tests  were  rec¬ 
ommended  for  inclusion  in  the  predictor  battery.  All 
required  modifications  before  they  were  deemed  ready 
for  administration.  Examples  of  the  recommended 
modifications  follow. 

•  Scan:  Increase  clariry  of  figures,  increase  number  of 
test  items,  and  possibly  use  mouse  to  decrease  keyboard 
skills  requirement. 

•  Letter  Facroiy:  Increase  planning/thinking  ahead  re¬ 
quirement  (e.g.,  by  adding  boxes  at  top  of  columns). 

•  Sound:  Investigate  possibility  of  changing  scoring  to 
allow  partial  credit  for  partially  correct  answers. 

•  Dial  Reading:  Increase  number  of  items,  decrease 
time  limit,  investigate  fineness  of  differentiation  re¬ 
quired. 

■  PEAQ:  Decrease  number  of  items  (focus  on  only 
critically  important  worker  requirements),  replace  ran¬ 
dom  response  items,  edit  response  options  for  all  items. 

•  Analogy:  Delete  information  processing  component, 
possibly  add  some  of  the  Classification  test  items. 

•  ATS,  TW,  SV:  Separate  individual  tests  from  PTS 
administration,  shorten  tests. 

Three  additional  tests  were  strongly  considered  for 
inclusion,  but  with  further  modifications:  Planes,  Pro¬ 
jection  (perhaps  modified  to  represent  above-ground 
stimuli),  and  a  numerical  ability  test.  The  latter  repre¬ 
sented  a  worker  requirement  that  was  otherwise  not 
measured  by  the  set  of  “included”  tests.  The  plan  for  the 
numerical  ability  test  was  initially  to  include  items 
modified  from  several  existing  tests:  Headings,  Direc¬ 
tion  and  Distance,  and  Time,  all  of  which  include 
components  of  on-the-job  numerical  computation. 
Angles  and  Dials  would  be  added  to  round  out  the 
numeric  ability  construct. 


22 


Possible  Gaps  in  Coverage 

Viewing  the  12  tests  on  the  preliminary  list  as  a 
whole,  the  coverage  of  worker  requirements  appeared 
quite  good.  However,  a  few  important  worker  require¬ 
ments  remained  unrepresented:  reading  comprehen¬ 
sion,  memory,  and  street  physics.  There  was  some 
discussion  about  including  measures  of  these  three 
requirements.  A  reading  test  could  be  prepared,  using 
very  short,  face  valid  passages,  where  the  passage  and  the 
test  question  could  be  displayed  on  screen  at  the  same 
time.  Discussions  about  adding  a  memory  test  primarily 
focused  on  a  modification  of  the  Map  test,  which  would 
require  candidates  to  indicate  whether  the  stimulus  had 
changed  or  remained  the  same  since  they  “memorized” 
it.  The  possibility  of  finding  a  published  test  measuring 
street  physics  was  also  discussed.  If  such  a  published  test 
could  not  be  found,  some  kind  of  mechanical  or  abstract 
reasoning  test  might  be  included  as  a  close  approximation. 

Excluded  Tests 

Three  of  the  20  tests  reviewed  were  deleted  from 
further  consideration:  Stix,  Map  (except  as  it  might  be 
revised  to  cover  memory),  and  Syllogism.  These  tests 
were  deleted  because  of  problems  with  test  construc¬ 
tion,  and/or  questionable  relevance  for  important  job 
requirements,  and/or  redundancy  with  the  included 
measures. 

Additional  Recommendations 

Several  additional  recommendations  were  made  con¬ 
cerning  the  predictor  battery  and  its  documentation. 
The  first  was  that  all  tests,  once  revised,  be  carefully 
reviewed  to  ensure  that  the  battery  adheres  to  good  test 
construction  principles  such  as  consistency  of  direc¬ 
tions  and  keyboard  use,  reading/ vocabulary  level,  and 
balancing  keyed  response  options. 

A  second  recommendation  was  that  linkages  be  pro¬ 
vided  for  worker  requirements  that  do  not  currently 
have  documented  linkages  with  ATCS  job  duties.  The 
current  documentation  (from  the  Job  Analysis  report) 
was  incomplete  in  this  regard. 

A  third  recommendation  was  to  “pilot  test”  the 
predictor  set  in  February  1997.  It  was  thought  that  this 
would  yield  the  kind  of  data  needed  to  perform  a  final 
revision  of  all  predictors,  select  the  best  test  items, 
shorten  tests,  reduce  redundancy  across  tests,  ensure 
clarity  of  instructions,  and  so  on. 


AT-SAT  ALPHA  BATTERY 

Based  on  the  reviews  and  recommendations  of  the 
expert  panel,  the  AT-SAT  researchers  developed  the 
predictor  battery  to  be  pilot  tested,  called  the  Alpha 
Battery.  It  consisted  of  14  tests  given  across  five  testing 
blocks.  They  were: 

Block  A:  Air  Traffic  Scenarios 
Block  B:  Sound  test  and  Letter  Factory  test 
Block  C:  Dials  test.  Static  Vector/Continuous  Memory 
test,  and  Experiences  Questionnaire  (formerly 
PEAQ) 

Block  D:  Time  Wall/Pattern  Recognition  test.  Analogy 
test,  and  Classification  test 

Block  E:  Word  Memory  test.  Scan  test.  Planes  test.  Angles 
test,  and  Applied  Mathematics  test 

A  short  description  of  the  tests  follows.  In  a  few 
instances,  details  reflect  modifications  made  in  the 
alpha  pilot  tests  for  use  in  the  beta  (validation)  testing. 

Air  Traffic  Scenarios  Test 

This  is  a  low-fidelity  simulation  of  an  air  traffic 
control  radar  screen  that  is  updated  every  7  seconds.  The 
goal  is  to  maintain  separation  and  control  of  varying 
numbers  of  simulated  aircraft  (represented  as  data  blocks) 
within  the  participant’s  designated  airspace  as  effi¬ 
ciently  as  possible.  Simulated  aircraft  either  pass  through 
the  airspace  or  land  at  one  of  two  airports  within  the 
airspace.  Each  aircraft  indicates  its  present  heading, 
speed,  and  altitude  via  its  data  block.  There  are  eight 
different  headings  representing  45-degree  increments, 
three  different  speed  levels  (slow,  moderate,  fast),  and 
four  different  altitude  levels  (l=lowest  and  4=highest). 

Separation  and  control  are  achieved  by  communicat¬ 
ing  and  coordinating  with  each  aircraft.  This  is  accom¬ 
plished  by  using  the  computer  mouse  to  click  on  the 
data  block  representing  each  aircraft  and  providing 
instructions  such  as  heading,  speed,  or  altitude.  New 
aircraft  in  the  participant’s  airspace  have  data  blocks 
appear  in  white  that  turn  green  once  the  participant  has 
communicated  with  them.  Rules  for  handling  aircraft 
are  as  follows:  (1)  maintain  a  designated  separation 
distance  between  planes,  (2)  land  designated  aircraft  at 
their  proper  airport  and  in  the  proper  landing  direction 
flying  at  the  lowest  altitude  and  lowest  speed,  (3)  route 
aircraft  passing  through  the  airspace  to  their  designated 
exit  at  the  highest  altitude  and  highest  speed.  The 
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version  of  ATST  that  was  incorporated  in  the  alpha 
battery  was  modified  to  operate  in  the  windows  envi¬ 
ronment  (Broach,  1996). 

Analogy  Test 

The  Analogy  test  measures  the  participant’s  ability  to 
apply  the  correct  rules  to  solve  a  given  problem.  An 
analogy  item  provides  a  pair  of  either  words  or  figures 
that  are  related  to  one  another  in  a  particular  way.  In  the 
analogy  test,  a  participant  has  to  choose  the  item  that 
completes  a  second  pair  in  such  a  way  that  the  relation¬ 
ship  of  the  items  (words  or  figures)  in  the  second  pair  is 
the  same  as  that  of  the  first. 

The  test  has  57  items:  30  word  analogies  and  27 
visual  analogies.  Each  item  has  five  answer  options.  The 
scoring  is  based  primarily  on  the  number  of  correct 
answers  and  secondarily  on  the  speed  with  which  the 
participant  arrived  at  each  answer.  Visual  analogies  can 
contain  either  pictures  or  figures.  The  instructions 
inform  the  participant  that  the  relationships  for  these 
two  types  ofvisual  analogies  are  different.  Picture  analo¬ 
gies  are  based  on  the  relationships  formed  by  the  mean¬ 
ing  of  the  object  pair  (e.g.,  relationships  of  behavior, 
function,  or  features).  Figure  analogies  are  based  on  the 
relationships  formed  by  the  structure  of  the  object  pair 
(e.g.,  similar  parts  or  rotation). 

Angles  Test 

The  Angles  test  measures  the  participant’s  ability  to 
recognize  angles.  This  test  contains  30  multiple-choice 
questions  and  allows  participants  up  to  8  minutes  to 
complete  them.  The  score  is  based  on  the  number  of 
correct  answers  (with  no  penalty  for  wrong  or  unan¬ 
swered  questions).  There  are  two  types  of  questions. 
The  first  presents  a  picture  of  an  angle,  and  the  pattici- 
pant  chooses  the  correct  answer  of  the  angle  (in  degrees) 
from  among  four  response  options.  The  second  presents 
a  measure  in  degrees,  and  the  participant  chooses  the 
angle  (among  four  response  options)  that  represents  that 
measure. 

Applied  Mathematics  Test 

This  test  contains  30  multiple-choice  questions  and 
allows  participants  up  to  21  minutes  to  complete  them. 
The  score  is  based  on  the  number  of  correct  answers 
(with  no  penalty  for  wrong  or  unanswered  questions). 
The  test  presents  five  practice  questions  before  the  test 
begins.  Questions  such  as  the  following  are  contained 
on  the  test: 


A  plane  has  flown  for  3  hours  with  a  ground  speed  of 
210  knots.  How  far  did  the  plane  travel? 

These  questions  require  the  participant  be  able  to 
factor  in  such  things  as  time  and  distance  to  identify  the 
correct  answer  from  among  the  four  answer  choices. 

Dials  Test 

The  Dials  test  is  designed  to  test  the  participant’s 
ability  to  quickly  identify  and  accurately  read  certain 
dials  on  an  instrument  panel.  The  test  consists  of  20 
items  completed  over  a  total  time  of  9  minutes.  Indi¬ 
vidual  items  are  self-paced  against  the  display  of  time 
left  in  the  test  as  a  whole.  Participants  are  advised  to  skip 
difficult  items  and  come  back  to  them  at  the  end  of  the 
test.  The  score  is  based  on  the  number  of  items  answered 
correctly.  The  test  scteen  consists  of  seven  dials  in  two 
rows,  a  layout  which  remains  constant  throughout  the 
test.  Each  of  the  seven  dials  contains  unique  flight 
information.  The  top  row  contains  the  following  dials: 
Voltmeter,  RPM,  Fuel-air  Ratio,  and  Altitude.  The 
bottom  row  contains  the  Amperes,  Temperature,  and 
Airspeed  dials. 

Each  test  item  asks  a  question  about  one  dial.  To 
complete  each  item,  the  participant  is  instructed  to  (1) 
find  the  specified  scale  on  the  instrument  panel;  (2) 
determine  the  point  on  the  scale  represented  by  the 
needle;  (3)  find  the  corresponding  value  among  the  five 
answer  options;  (4)  use  the  numeric  keypad  to  press  the 
number  corresponding  to  the  option. 

Experiences  Questionnaire 

The  Experiences  Questionnaire  assesses  whether  par¬ 
ticipants  possess  certain  work-related  attributes  by  ask¬ 
ing  questions  about  past  experiences.  There  are  201 
items  to  be  completed  in  a  40-minute  time  frame.  Items 
cover  attitudes  toward  work  relationships,  rules,  deci¬ 
sion-making,  initiative,  ability  to  focus,  flexibility,  self- 
awareness,  work  cycles,  work  habits,  reaction  to  pressure, 
attention  to  detail,  and  other  related  topics.  Each  ques¬ 
tion  is  written  as  a  statement  about  the  participant’s  past 
experience  and  the  participant  is  asked  to  indicate  their  level 
of  agreement  with  each  statement  on  the  following  5-point 
scale:  1=  Definitely  true,  2=  Somewhat  true,  3=  Neither 
tme  nor  false,  4=  Somewhat  false,  5=  Definitely  false. 

Letter  Factory  Test 

This  test  simulates  a  factory  assembly  line  that  manu¬ 
factures  letters  A  to  D  of  the  alphabet.  Examinees 
perform  multiple  and  often  concurrent  tasks  during  the 


24 


test  with  aid  of  a  mouse.  Tasks  include:  (1)  picking  up 
letters  of  various  colors  from  a  conveyor  belt  and  loading 
them  into  boxes  of  the  same  color;  (2)  moving  empty 
boxes  from  storage  to  the  loading  area;  (3)  ordering  new 
boxes  when  supplies  become  low;  (4)  calling  Quality 
Control  when  defective  letters  appear;  and  (5)  answer¬ 
ing  multiple-choice  questions  about  the  factory  floor 
display.  The  test  is  comprised  of  1 8  test  parts;  each  part 
begins  when  the  letters  appear  at  the  top  of  the  belts  and 
ends  with  four  multiple-choice  questions.  Awareness 
questions  assess  the  state  of  the  screen  display.  Easier 
questions  are  presented  during  lulls  in  assembly  line 
activity  and  assess  the  current  state  of  the  display.  More 
difficult  questions  are  asked  during  peak  activity  and 
assess  what  a  future  display  might  look  like. 

Overall  scores  on  the  LFT  are  based  on  (1)  the 
number  of  boxes  correctly  moved  to  the  loading  area; 

(2)  the  time  it  takes  to  move  a  box  after  it  is  needed;  (3) 
the  number  of  letters  correctly  placed  into  boxes;  and  (4) 
answers  to  the  awareness  questions.  The  following 
actions  lower  test  scores:  (1)  allowing  letters  to  fall  off 
the  end  of  a  belt;  (2)  placing  letters  in  an  incorrect  box; 

(3)  not  moving  a  box  into  the  loading  area  when  needed; 
and  (4)  attempting  to  move  the  wrong  box  into  the 
loading  area. 

Planes  Test 

The  Planes  test  contains  three  parts,  each  with  48 
items  to  be  completed  in  6  minutes.  Each  individual 
item  must  be  answered  within  12  seconds.  Part  1: 
Participants  perform  a  single  task.  Two  planes  move 
across  a  screen;  one  plane  is  red,  the  other  is  white.  Each 
plane  moves  toward  a  “destination”  (a  vertical  line)  at  a 
different  speed.  The  planes  disappear  before  they  reach 
their  destinations,  and  the  participant  must  determine 
which  plane  would  have  reached  its  destination  first.  To 
answer  each  item,  the  participant  presses  the  “red”  key 
if  the  red  plane  would  have  reached  the  destination  first, 
and  the  “white”  key  if  the  white  plane  would  have 
arrived  first.  Participants  can  answer  while  the  planes  are 
still  moving,  or  shortly  after  they  disappear.  Part 2:  Part 
2  is  similar  to  Part  1 ,  but  participants  must  now  perform 
two  tasks  at  the  same  time.  In  this  part  of  the  test, 
participants  determine  which  of  two  planes  will  arrive  at 
the  destination  first.  Below  the  planes,  a  sentence  will 
appear  stating  which  plane  will  arrive  first.  The  partici¬ 
pant  must  compare  the  sentence  to  their  perception  of 
the  planes’  arrival,  and  press  the  “true”  key  to  indicate 
agreement  with  the  statement,  or  the  “false”  key  to 
indicate  disagreement.  Part  3:  Participants  perform  the 


same  tasks  as  in  Part  2,  but  the  statements  below  the 
planes  are  a  little  more  difficult  to  analyze.  In  all  other 
respects,  the  participants  perform  in  the  same  manner. 

Scan  Test 

In  the  Scan  test,  participants  monitor  a  field  that 
contains  discrete  objects  (called  data  blocks)  which  are 
moving  in  different  directions.  Data  blocks  appear  in 
the  field  at  random,  travel  in  a  straight  line  for  a  short 
time,  then  disappear.  During  the  test,  the  participant 
sees  a  blue  field  that  fills  the  screen,  except  for  a  1-inch 
white  bar  at  the  bottom.  In  this  field,  up  to  1 2  green  data 
blocks  may  be  present.  The  data  blocks  each  contain  two 
lines  of  letters  and  numbers  separated  by  a  horizontal 
line.  The  upper  line  is  the  identifier  and  begins  with  a 
letter  followed  by  a  2-digit  number.  The  lower  line 
contains  a  3-digit  number.  Participants  are  scored  on 
the  speed  with  which  they  notice  and  respond  to  the  data 
blocks  that  have  a  number  on  the  lower  line  outside  a 
specified  range.  Throughout  the  test,  this  range  is  dis¬ 
played  at  the  bottom  of  the  screen  (e.g.,  360-710).  To 
“respond”  to  a  data  block,  the  participant  types  the  2- 
digit  number  from  the  upper  line  of  the  block  (ignoring 
the  letter  that  precedes  it),  then  presses  “enter.” 

Sotmd  Memory  Test 

The  Sound  Memory  test  measures  a  participant’s 
listening  comprehension,  memory,  and  hand-eye  coor¬ 
dination.  Participants  must  hear,  remember,  and  record 
strings  of  numbers  varying  in  length  from  5  to  10  digits. 
After  the  digits  have  been  read,  there  is  a  brief  pause. 
Then  a  yellow  box  will  appear  on  screen,  and  partici¬ 
pants  must  type  in  the  numbers  they  heard  and  remem¬ 
bered,  in  the  order  presented  orally.  Participants  may 
use  the  backspace  to  delete  and  correct  the  numbers  they 
enter,  and  press  the  “enter”  key  to  submit  the  answer. 

Each  participant’s  score  equals  the  total  number  of 
digits  the  participant  remembers  correctly.  If  the  partici¬ 
pant  transposes  two  digits  then  half-credit  is  given. 
Items  must  be  answered  in  the  order  presented — par¬ 
ticipants  cannot  skip  and  return  to  previous  items.  If  too 
few  digits  are  typed  then  the  missing  digits  are  scored  as 
incorrect;  if  too  many  digits  are  typed  then  the  extra 
digits  are  ignored.  The  object  is  simply  to  recall  digits 
heard  in  the  correct  order. 

Time  Wall/Pattern  Recognition  Test 

The  Time  Wall/Pattern  Recognition  test  consists  of 
two  tasks  that  measure  the  examinee’s  ability  to  judge 
the  speed  of  objects  and  to  compare  visual  patterns  at  the 
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same  time.  In  the  time  judgment  task,  the  participant 
watches  a  square  move  from  left  to  right  and  estimates 
when  it  will  hit  a  wall  positioned  on  the  right  side  of  the 
display  screen.  In  the  pattern  comparison  task,  the 
participant  determines  whether  two  patterns  are  the 
same  or  different  from  each  other.  Each  exercise  begins 
with  a  square  moving  toward  a  wall  at  a  fast,  medium, 
or  slow  speed.  After  a  short  while,  the  square  disappears 
behind  the  pattern  recognition  screen.  The  participant 
must  hit  the  stop  key  at  the  exact  moment  the  square  hits 
the  wall. 

In  the  pattern  comparison  task,  the  participant  is 
shown  two  blue  circles,  each  with  an  overlay  pattern  of 
white  dots.  Test  takers  are  requested  to  press  the  “same” 
key  if  the  patterns  are  the  same  or  press  the  “differ”  key 
if  the  patterns  are  different.  Concurrently,  participants 
should  press  the  “stop”  key  when  they  think  the  square 
will  hit  the  wall,  even  if  they  are  in  the  middle  of 
comparing  two  patterns.  Participants  are  scored  upon 
how  quickly  they  respond  without  making  mistakes. 
The  score  is  lowered  for  each  incorrect  judgment. 

Word  Memory  Test 

The  Word  Memory  test  presents  a  series  of  24  words 
in  an  artificial  language  (i.e.,  “SPRON”)  and  their 
associated  English  equivalents.  The  goal  is  to  memorize 
the  24  SPRON  words  and  their  English  equivalents  and 


then  recall  these  at  two  different  testing  times:  one 
immediately  following  a  practice  session  and  another  in 
a  subsequent  testing  block.  The  practice  session  lasts  4 
minutes,  during  which  the  list  of  24  SPRON  words  and 
their  English  equivalents  are  displayed  in  a  box  to  the 
right  of  the  display  screen  while  the  multiple-choice 
items  are  displayed  on  the  left.  The  practice  items  allow 
the  test  takers  to  apply  their  memory  by  allowing  them 
to  review  the  SPRON-English  list  of  words  as  a  refer¬ 
ence.  The  first  testing  session  starts  immediately  follow¬ 
ing  the  practice  session  and  lasts  5  minutes.  The  second 
testing  session  starts  in  a  subsequent  testing  block  (after 
a  break  time)  and  also  lasts  for  5  minutes.  Each  multiple- 
choice  item  displays  the  SPRON  word  as  the  item  stem 
and  displays  five  different  English  equivalents  as  the  five 
response  alternatives. 

Conclusion 

The  initial  AT-SAT  test  battery  (Alpha)  was  profes¬ 
sionally  developed  after  a  careful  consideration  of  mul¬ 
tiple  factors.  These  included  an  examination  of  the 
SACHA  job  analysis  and  prior  job  analyses  that  pro¬ 
duced  lists  of  worker  requirements,  prior  validation 
research  on  the  ATCS  job,  and  the  professional  judg¬ 
ment  of  a  knowledgeable  and  experienced  team  of 
testing  experts. 
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CHAPTER  3.2 


Air  Traffic  -  Selection  and  Training  Alpha  Pilot  Trial  After-Action  Report 

Claudette  Archambault,  Robyn  Harris 
Caliber  Associates 


INTRODUCTION 

The  purpose  of  this  report  is  to  document  the  obser¬ 
vations  of  the  Air  Traffic  -  Selection  and  Training 
Completion  (AT-SAT)  predictor  battery  (alpha  ver¬ 
sion)  pilot  trial.  The  AT -SAT  predictor  batteiy  is  a  series 
of  tests  in  five  blocks  (A  through  E)  of  90  minutes  each 
and  four  different  ending  blocks  of  20  minutes  each. 
The  pilot  test  was  administered  February  19  through 
March  2, 1997,  in  the  Air  Traffic  Control  School  at  the 
Pensacola  Naval  Air  Station  in  Pensacola,  Florida.  Par¬ 
ticipants  consisted  of  566  students  stationed  at  the 
Naval  AirTechnical  T  raining  Center  (NATTC).  Of  the 
566  students,  215  of  the  participants  were  currently 
enrolled  in  the  Air  T raffic  Control  School  and  346  were 
students  waiting  for  their  classes  at  NATTC  to  begin. 
(The  status  of  five  participants  was  unknown.) 

This  report  contains  the  following  sections: 

•  Pilot  Test  Description  and  Procedures 

•  General  Observations 

•  Feedback  on  Test  Block  A 

•  Feedback  on  Test  Block  B 

•  Feedback  on  Test  Block  C 

•  Feedback  on  Test  Block  D 
■  Feedback  on  Test  Block  E 

•  Feedback  on  the  Ending  Block 

The  report  concludes  with  a  summary  of  all  of  the 
feedback  and  observations. 

The  At-SAT  Pilot  Test  Description  and 
Administration  Procedures 

The  following  sections  describe  the  AT-SAT  pilot 
test  and  pilot  test  administration  procedures. 


AT-SAT  Pilot  Test  Description 

The  AT-SAT  Pilot  Test  is  a  series  of  five  test  blocks 
(Blocks  A  through  E)  and  Ending  Blocks.  (There  are 
four  different  Ending  Blocks.)  The  tests  are  designed  to 
measure  different  aptitudes  required  for  successfully 
performing  the  job  of  air  traffic  controller.  Tests  are 
subdivided  as  follows: 

■  Block  A  contains  one  test  entitled  Air  Traffic  Sce¬ 
narios  (ATS). 

■  Block  B  contains  the  Sound  Test  and  the  Letter 
Factory  Test  (LFT). 

•  Block  C  contains  the  Dials  Test,  Static  Vector/Con¬ 
tinuous  Memory  T est  (SV CM) ,  and  Experiences  Ques¬ 
tionnaire. 

•  Block  D  contains  the  Time  Wall/Pattern  Recogni¬ 
tion  Test  (TWPR),  the  Analogy  Test,  and  the  Classifi¬ 
cation  Test. 

•  Block  E  contains  the  Word  Memory  Test,  the  Scan 
Test,  the  Planes  Test,  the  Angles  Test,  and  the  Applied 
Mathematics  Test. 

Depending  on  the  Participant’s  group  number,  the 
Ending  Block  consisted  of  one  of  the  following. 

■  the  LFT 

■  the  ATS 

•  the  SVCM  and  Word  Memory  tests 

•  the  Word  Memory  and  TWPR  tests 

The  following  section  describes  the  test  administra¬ 
tion  procedures  including  the  sequence  of  the  testing 
blocks  for  groups  of  participants. 

Pilot  Test  Administration  Procedures 

Participants  were  arranged  in  five  groups  of  ten 
(Groups  1  through  5).  Test  Administrators  (TAs)  sup¬ 
plied  the  testing  rooms  with  55  computers.  Fifty  of  the 
computers  were  used  for  testing  stations;  five  were 
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failure-safe  or  recovery  stations.  Recovery  stations  were 
reserved  for  use  by  participants  when  TAs  were  not  able 
to  restore  operation  to  a  malfunctioning  computer. 

In  one  classroom,  there  were  33  computers  (30  for 
testing  and  three  failure-safe  computers):  Groups  1,  2, 
and  3  were  tested  on  computers  1  through  30  (See 
Exhibit  3.2.1).  In  a  second  classroom,  there  were  22 
computers  (20  for  testing  and  two  failure-safe  comput¬ 
ers).  Groups  4  and  5  were  tested  in  the  second  room  on 
computer  numbers  31  through  50.  Exhibit  3.2.1  dis¬ 
plays  the  sequencing  of  test  blocks.  (The  exhibit  does 
not  reflect  breaks.) 

Participants  were  offered  a  minimum  of  a  ten-minute 
break  between  each  of  the  five  testing  sections.  Because 
the  tests  are  self-paced,  participants  were  not  required  to 
take  the  1 0-minute  breaks  between  blocks.  They  were 
required  to  take  a  1 .5  hour  meal  break  between  sessions 
two  and  three. 

GENERAL  OBSERVATIONS 

This  section  presents  some  general  observations  about 
the  entire  AT-SAT  Battery  Pilot  Test.  The  remarks  in 
this  section  address  the  instructions,  the  test  ending,  the 
purpose  of  the  tests,  and  the  introductory  block. 

Instructions 

Instructions  for  several  of  the  tests  in  the  battery  need 
improved  clarity.  Participants  often  did  not  understand 
the  test  instructions  as  written  but  proceeded  with  the 
tests,  anticipating  that  the  objective  of  the  tests  would 
become  more  clear  as  the  tests  proceeded.  Too  often, 
however,  participants  still  did  not  understand  the  objec¬ 
tive  even  after  attempting  a  few  examples.  (After  partici¬ 
pants  completed  the  examples,  they  would  often  raise 
their  hand  and  ask  for  further  instructions.)  Therefore, 
any  practice  sessions  for  the  tests  did  not  clarify  the 
confusing  instructions.  The  test  instructions  that  need 
revision  and  feedback  for  specific  test  blocks  are  dis¬ 
cussed  in  the  following  sections. 

Purpose  of  the  Tests 

Participants  also  required  further  clarification  of  the 
purpose  of  tests  within  the  blocks  during  the  practice 
session  (instead  of  before  or  after  the  test).  Perhaps  a 
short  paragraph  including  the  aptitudes  that  are  being 
tested  would  clarify  the  purpose  of  certain  tests. 

In  addition  to  more  specific  test  instructions,  an 
introductory  screen  at  the  start  of  each  block,  to  include 
the  number  of  different  tests  within  the  specific  block. 


the  names  of  the  tests  and  a  short  description  of  each 
test,  the  aptitudes  that  are  being  tested,  and  the  time 
allotted  for  each  test  should  be  added.  This  screen  may 
eliminate  discrepancies  where  participants  are  unclear 
as  to  whethet  to  continue  with  the  other  tests  in  the 
block  when  they  reach  the  end  of  a  test. 

Test  Ending 

The  end  of  each  test  should  include  a  brief  statement 
in  a  text  box  stating  that  the  participant  has  completed 
the  current  test  and  should  press  enter,  or  click  on  the 
continue  button  (with  the  mouse  pointet)  to  proceed  to 
the  next  test  in  the  block.  The  text  box  could  also  state 
the  number  of  tests  completed  and  the  number  of  tests 
that  remain  for  each  block. 

Currently,  some  blocks  do  not  indicate  the  end  of  the 
block  with  a  text  box.  Some  tests  simply  go  to  a  blue 
screen  and  do  not  indicate  that  tbe  test  bas  indeed 
ended.  The  final  test  in  a  block  should  indicate  not  only 
that  the  current  test  is  finished  but  also  that  the  partici¬ 
pant  has  indeed  completed  all  tests  within  the  block  and 
that  they  should  raise  their  hand  to  speak  with  the  Test 
Administrator. 

Not  all  of  the  tests  indicate  the  results  of  the  tests 
and/or  practice  sessions.  For  consistency,  either  all 
tests  should  display  results,  or  all  tests  should  not 
display  results. 

Introductory  Block 

The  addition  of  an  Introductory  Block  (IB)  is  recom¬ 
mended.  The  IB  could  explain  of  the  general  purpose  of 
the  testing  a  modified  vetsion  of  the  Keyboard  Familiar¬ 
ization  section  and  the  current  Background  Informa¬ 
tion  questions. 

The  explanation  of  the  general  purpose  of  the  test 
might  also  include  a  brief  description  of  the  evolution  of 
the  test  (how  the  FAA  came  to  design  this  specific  testing 
procedure).  This  section  could  describe  the  types  of  tests 
and  the  general  purpose  of  the  tests  (i.e.,  ability  to  multi¬ 
task,  ability  to  follow  instructions,  skill  with  plane 
routing  procedures,  etc.).  Finally,  general  grading/scor¬ 
ing  procedures  could  be  explained  with  more  specific 
explanations  within  each  of  the  tests. 

The  Keyboard  Familiarization  (KF)  currently  in¬ 
cludes  insttuction  and  practice  for  the  number  keys  and 
the  “A,  B,  C”  keys  (after  the  Test  Administtator  ex¬ 
changes  the  slash,  star,  and  minus  keys  with  the  A,  B, 
and  C  keys)  on  the  numeric  pad  on  the  right  side  of  the 
keyboard.  Instructions  should  be  modified  to  include 
the  names  of  the  tests  requiring  the  use  of  these  keys. 
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Directions  under  BCF  should  also  include  the  names 
of  the  tests  that  will  require  the  use  of  the  numerical  keys 
on  the  top  of  the  keyboard.  A  practice  session  should 
also  be  included  for  these  keys  to  allow  participants  to 
become  acquainted  with  the  placement  of  their  hands  at 
the  top  of  the  keyboard. 

Background  information  questions  should  be  in¬ 
cluded  within  the  Introductory  Block.  This  will  allow 
participants  to  practice  using  the  keyboard  outside  of  a 
testing  block.  It  will  also  allow  them  to  ask  the  Test 
Administrator  questions  about  the  test,  use  of  the  key¬ 
board,  the  placement  of  hands  on  the  keyboard,  and  so  on. 

FEEDBACK  ON  TEST  BLOCK  A 

Block  A  was  the  only  block  that  consisted  of  only  one 
test.  Therefore,  the  comment  below  applies  to  the  Air 
Traffic  Scenarios  Test  (ATST). 

The  ATST  requires  participants  to  manipulate  the 
heading,  speed,  and  level  (or  altitude)  of  planes  in  their 
airspace.  On  the  testing  screen,  participants  see  the 
airspace  for  which  they  are  responsible,  two  airports  for 
landing  planes,  and  four  exits  for  routing  planes  out  of 
the  airspace.  The  screen  also  displays  the  controls  for 
directing  planes:  (1)  the  heading  (to  manipulate  indi¬ 
vidual  plane  direction),  (2)  the  speed  (slow,  medium,  or 
fast),  (3) 

the  level  (1,  2,  3,  4).  Finally,  a  landing  heading 
indicator  is  displayed  that  informs  the  participant  of  the 
direction  to  land  planes  at  each  of  the  airports. 

Instructions 

Instructions  for  the  ATST  may  need  more  clarification. 
Often,  pardcipants  required  further  clarification  on: 

•  the  meaning  of  each  of  the  plane  descriptors  that 
appear  on  the  screen 

•  the  difference  between  white  and  green  planes 

•  the  need  to  click  on  the  graphic  (depicted  as  an  arrow) 
that  represents  the  plane  (versus  the  text  descriptors  of 
the  plane)  to  change  the  heading,  level,  and  speed. 

Instructions  need  to  be  rewritten  to  include  more 
details  on  the  descriptors  accompanying  the  planes. 
Perhaps  in  the  instructions  section,  the  descriptors  can 
be  enlarged  on  the  screen  with  an  arrow  pointing  to  the 
definition  of  the  letters  and  number  as  in  Exhibit  3.2.2. 


New  Planes 

New  planes  that  appear  in  the  participant’s  airspace 
are  white  (while  all  other  planes  are  green).  The  white 
planes  remain  circling  at  the  point  where  they  entered 
the  airspace  until  they  receive  acknowledgment  from 
the  controller  (by  clicking  on  the  graphic  with  the 
mouse  pointer).  Often  during  testing  participants  did 
not  understand  the  purpose  of  the  white  planes  in  their 
airspace.  They  would  leave  the  white  planes  circling  and 
never  manipulate  their  heading,  speed,  or  level.  White 
planes  need  to  be  more  clearly  defined  as  new  planes  in 
the  controller’s  airspace  that  require  acknowledgment 
by  the  controller. 

Countdown 

At  the  start  of  a  scenario,  participants  often  did  not 
notice  the  countdown  (on  the  counter  at  the  bottom 
right-hand  corner  of  the  screen)  before  the  beginning  of 
a  test.  There  is  a  delay  (of  approximately  7  seconds) 
between  the  time  the  test  initially  appears  on  the  screen 
and  the  time  the  participant  can  perform  an  action  to  the 
planes  on  the  screen. 

During  this  delay,  some  participants  continuously 
pushed  the  “enter”  button,  which  would  often  result  in 
one  of  two  consequences:  (1)  The  computer  screen 
would  permanently  freeze  (such  that  the  system  would 
need  to  be  rebooted);  (2)  at  the  end  of  the  test,  the 
participant  received  the  plain  blue  screen  (indicating 
that  the  test  was  complete).  However,  once  the  Test 
Administrator  closed-out  the  blue  screen  and  returned 
to  the  program  manager,  there  would  remain  a  row  of 
several  icons  with  each  icon  indicating  an  air  traffic 
scenario.  The  Test  Administrator  would  need  to  pre¬ 
sume  that  the  next  correct  test  in  the  sequence  was  the 
first  in  the  row  of  icons  and  double  click  on  that  icon  to 
begin  a  scenario.  At  the  end  of  each  scenario,  the  Test 
Administrator  would  double-click  on  the  next  scenario 
in  the  row  of  icons  until  all  scenarios  were  complete. 

For  participants  to  clearly  see  that  there  is  a  delay 
before  they  can  manipulate  the  planes  on  the  screen, 
perhaps  the  countdown  timer  can  be  moved  to  a  more 
conspicuous  position  in  the  middle  of  the  screen  (as  in 
the  Static  Vector/Continuous  Memory  Test).  An  alter¬ 
native  would  be  to  display  the  counter  in  a  brightly 
colored  text  box  (still  in  the  bottom  right-hand  corner 
of  the  screen).  After  the  countdown  timer  had  finished, 
the  text  box  could  change  colors  and  blend  with  the 
other  instructions  on  the  screen. 
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Landing  Heading  Indicator 

Participants  often  did  not  notice  the  landing  heading 
indicator  located  on  the  bottom  right-hand  corner  of 
the  screen.  Others  noticed  the  arrow  but  did  not  under¬ 
stand  its  purpose.  Further  instruction  on  the  location 
and  the  purpose  of  the  landing  heading  indicator  may  be 
necessary.  Perhaps  during  the  practice  scenario,  a  text 
box  can  flash  on  the  screen  pointing  out  when  the 
participant  has  landed  a  plane  in  the  incorrect  direction. 
The  same  idea  may  be  useful  to  point  out  other  partici¬ 
pant  errors  during  the  practice  session(s). 

FEEDBACK  ON  TEST  BLOCK  B 

This  section  details  the  observations  for  the  two  tests 
in  Block  B:  the  Sound  Test  and  the  Letter  Factory  Test. 
Specific  comments  about  each  test  are  provided  below. 

Sound  Test 

For  this  test,  the  participant  uses  headphones  to  listen 
to  a  sequence  of  numbers.  Then  the  participant  must 
repeat  the  sequence  of  the  numbers  heard  using  the 
right-hand  numeric  keypad  to  record  the  sequence  of 
numbers. 

Failures 

It  was  found  in  the  first  day  of  testing  that  computers 
would  lock  or  fail  if  the  Sound  Test  was  run  after  any 
other  blocks.  In  other  words,  unless  Block  B  was  first  in 
the  sequence  of  testing,  computers  would  fail  (at  the 
moment  participants  are  prompted  for  sound  level 
adjustment)  and  need  to  be  rebooted.  This  proved 
disruptive  to  other  participants  and  delayed  the  start  of 
the  test  (since  Test  Administrators  can  only  aid  one  or 
two  participants  at  a  time).  To  prevent  failures  during 
the  testing.  Test  Administrators  would  reboot  every 
computer  before  the  start  of  Block  B.  Still,  the  software 
would  sometimes  fail  at  the  moment  the  participant  is 
requested  to  adjust  the  volume  to  their  headphones  via 
the  keyboard  (versus  the  sound  level  adjustment  located 
directly  on  the  headphones).  On  occasion,  the  Sound 
test  would  still  fail,  but  after  again  rebooting  the  com¬ 
puter,  the  program  recovered. 

After  several  attempts  to  restore  the  program  where 
there  were  repeated  failures,  the  computer  still  did  not 
allow  the  participant  to  continue  with  the  test.  In  these 
cases  where  a  computer  failed  repeatedly,  participants 


would  be  moved  to  a  failure-safe  computer.  It  is  likely 
that  such  failures  are  the  result  of  the  hardware  or 
hardware  configuration,  rather  than  the  software. 

There  is  another  possible  reason  for  the  failure  of  the 
Sound  Test.  Some  participants  would  attempt  to  re¬ 
peatedly  adjust  the  volume  of  their  headsets  with  the 
numbers  on  tbe  top  of  the  keyboard  rather  than  using 
the  number  keys  on  the  right-hand  side  of  the  keyboard 
(as  instructed).  It  is  possible  that  the  use  of  these  keys 
caused  some  of  the  failures. 

Removal  of  Headphones 

Upon  completion  of  the  Sound  Test,  participants 
often  keep  the  headphones  on  their  ears  throughout  the 
second  test  in  Block  B.  The  addition  of  some  text  at  the 
end  of  the  test  to  instruct  participants  to  remove  their 
headphones  might  be  useful. 

Letter  Factory  Test 

This  test  measures  four  abilities  required  to  perform 
air  traffic  controller  jobs.  These  abilities  are:  (1)  plan¬ 
ning  and  deciding  what  action  to  take  in  a  given 
situation  through  the  application  of  specific  rules;  (2) 
thinking  ahead  to  avoid  problems  before  they  occur;  (3) 
continuing  to  work  after  being  interrupted;  and  (4) 
maintaining  awareness  of  the  work  setting. 

Test  Instruction 

The  test  instructions  are  clear  and  well-written.  Few 
participants  had  questions  in  reference  to  the  tasks  they 
were  to  perform  once  the  test  began. 

Demonstration 

Participants  were  often  confused  during  the  demon¬ 
stration  because  the  pointer  would  move  when  they 
moved  the  mouse,  but  they  could  not  “click”  and 
manipulate  the  screen.  Participants  would  ask  Test 
Administrators  if  they  had  already  begun  the  test  since 
they  could  move  the  pointer.  Perhaps  the  mouse  can  be 
completely  disabled  during  the  demonstration  to  elimi¬ 
nate  confusion.  Disabling  the  mouse  would  allow  par¬ 
ticipants  to  concentrate  on  the  instructions  since  they 
would  not  be  distracted  by  movement  of  the  mouse. 

Mouse  Practice  Instructions 

Instructions  for  the  mouse  practice  session  are  not 
clear.  The  objective  of  the  mouse  practice  is  for  the 
participant  to  click  on  the  red  box  in  the  middle  of  the 
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screen  and  then  click  on  the  conveyer  belt  that  illumi¬ 
nates.  Participants  are  often  unsure  of  the  objective. 
Perhaps  text  box  instructions  can  be  displayed  on  the 
screen  that  direct  the  participant  to  click  on  the  red  box. 
As  the  participant  clicks  on  the  red  box,  another  instruc¬ 
tion  screen  would  appear,  telling  the  participant  to  click 
on  the  illuminated  conveyer  belt.  After  a  few  sequences 
with  text  box  instruction,  the  instructions  could  be 
dropped. 

Some  participants  had  difficulty  completing  the 
mouse  practice  session.  They  continuously  received 
messages  instructing  them  to  “...move  the  mouse  faster 
and  in  a  straight  line.”  Perhaps  there  should  be  a  limit  to 
the  number  of  mouse  practice  exercises.  It  is  possible 
that  some  participants  are  not  capable  of  moving  the 
mouse  quickly  enough  to  get  through  this  section. 

FEEDBACK  ON  TEST  BLOCK  C 

This  section  details  the  observations  and  suggestions 
for  the  three  tests  in  Block  C:  the  Dial  Test,  Static 
Vector/Continuous  Memory  Test,  and  Experiences 
Questionnaire.  Specific  comments  about  each  test  are 
provided  below. 

Dial  Test 

This  measures  the  participant’s  ability  to  quickly  and 
accurately  read  dials  on  an  instrument  panel.  Partici¬ 
pants  did  not  appear  to  have  difficulties  with  this  test. 
Test  Administrators  rarely  received  questions  from  par¬ 
ticipants  about  this  test. 

Static  Vector/Continuous  Memory  Test 

This  measures  the  participant’s  ability  to  perform 
perceptual  and  memory  tasks  at  the  same  time.  The 
perceptual  task  involves  determining  whether  two  planes 
are  in  conflict.  The  memory  task  involves  remembering 
flight  numbers.  On  each  trial,  the  screen  displays  a  plane 
conflict  problem  on  the  left  side  of  the  screen  and  a 
memory  problem  on  the  right  side  of  the  screen.  An 
attention  director  indicates  which  problem  the  partici¬ 
pant  is  to  work  on  and  is  located  in  the  middle  at  the 
bottom  of  the  screen. 

Instructions 

Participants  do  not  understand  the  instructions  for 
the  Memory  part  of  the  test.  Numerous  participants 
asked  for  clarity  on  what  numbers  they  were  to  compare. 


They  often  think  they  should  be  comparing  the  two 
numbers  that  are  on  the  screen  at  that  moment,  rather 
than  comparing  the  top  number  of  the  current  screen  to 
the  bottom  number  of  the  previous  screen. 

The  example  for  determining  the  conflict  for  the 
Static  Vector  questions  is  not  clear.  The  rule  about  the 
planes  requiring  2000  feet  difference  was  confusing 
because  they  did  not  understand  that,  although  the 
altitude  is  actually  displayed  in  hundreds  of  feet,  the 
altitude  represents  thousands  of  feet. 

Keyboard  Issues 

Many  participants  attempted  to  use  the  numerical 
keys  on  the  right-hand  side  of  the  keyboard  to  answer 
the  items  rather  than  the  using  the  keys  on  the  top  of  the 
keyboard  as  instructed.  When  participants  use  the  right- 
hand  keypad,  their  answers  are  not  recorded.  The  keys 
to  be  used  for  this  test  need  to  be  stated  more  explicitly. 

Participants  may  be  using  the  right-hand  keypad 
because  of  the  instruction  they  receive  in  the  Keyboard 
Familiarization  (KF)  section  at  the  beginning  of  the 
testing.  The  current  version  of  the  KF  only  provides 
instruction  for  use  of  the  keypads  on  the  right-hand  side 
of  the  keyboard.  The  KF  does  not  instruct  participants 
on  the  use  of  the  numerical  keys  on  the  top  of  the 
keyboard. 

As  noted  previously,  the  KF  needs  to  be  modified  to 
include  instructions  on  the  use  of  the  keys  on  the  top  of 
the  keyboard.  For  data  to  be  properly  collected,  it  is 
critical  for  participants  to  use  the  top  keys. 

Experiences  Questionnaire 

The  Experiences  Questionnaire  determines  whether 
the  participant  possesses  work-related  attributes  needed 
to  be  an  air  traffic  controller.  Participants  generally  did 
not  ask  any  questions  about  the  Experiences  Question¬ 
naire.  The  occasional  inquiry  was  in  reference  to  the 
purpose  of  certain  questions.  Test  Administrators  did 
not  receive  questions  about  the  wording  of  the  items. 

FEEDBACK  ON  BLOCK  D 

This  section  details  the  observations  and  suggestions 
for  the  three  tests  in  Block  D:  the  Time  Wall/Pattern 
Recognition  Test;  the  Analogy  Test;  and  the  Classifica¬ 
tion  Test.  Specific  comments  about  each  test  are  pro¬ 
vided  below. 
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Time  Wall/Pattern  Recognition  Test 

This  test  measures  the  participant’s  ability  to  judge 
time  and  motion  and  make  perceptual  judgments  at  the 
same  time.  The  time  judgment  task  involves  watching 
a  ball  move  (from  the  far  left-hand  side  of  the  screen) 
and  estimating  when  it  will  hit  a  wall  (located  on  the  far 
right  of  the  screen).  The  pace  of  the  ball  is  different  for 
every  scenario.  The  perceptual  task  involves  determin¬ 
ing  whether  two  patterns  are  the  same  or  different. 
These  tasks  must  be  performed  concurrently  by  the 
participant.  The  following  paragraphs  provide  observa¬ 
tions  and  suggestions  for  this  test.  This  section  includes 
observations  and  suggestions  for  improving  the  Time 
Wall/Pattern  Recognition  test  in  Block  D. 

Location  of  the  Broken  Wall 

When  the  participant  does  not  stop  the  ball  from 
hitting  the  wall  in  a  timely  manner,  the  screen  displays 
a  broken  wall.  However,  the  broken  wall  appears  in  the 
middle  of  the  screen,  rather  than  on  the  right-hand  side 
of  the  screen.  In  reference  to  this,  participants  often 
asked  how  they  were  to  determine  when  the  ball  would 
hit  the  wall  if  the  wall  was  always  moving.  Test  Admin¬ 
istrators  had  to  explain  that  the  wall  did  not  move,  but 
that  once  the  ball  broke  through  the  wall,  the  screen 
displayed  the  distance  past  the  wall  the  ball  had  moved. 
To  eliminate  confusion,  perhaps  the  broken  wall  can 
remain  on  the  right-hand  side  of  the  screen  and  just 
appear  broken  rather  than  being  moved  to  the  center  of 
the  screen. 

Keyboard 

As  with  the  Static  Vector/Continuous  Memory  Test, 
many  participants  attempted  to  use  the  numerical  keys 
on  the  right-hand  side  of  the  keyboard  to  answer  the 
items  rather  than  using  the  keys  on  the  top  of  the 
keyboard  as  instructed.  When  participants  use  the  right- 
hand  keypad,  their  answers  are  not  recorded.  The  keys 
to  be  used  for  this  test  need  to  be  stated  more  explicitly. 

Participants  may  be  using  the  keypad  because  of  the 
instruction  they  receive  in  the  Keyboard  Familiarization 
(KF)  section  at  the  beginning  of  the  first  block  of  testing. 
The  current  version  of  the  KF  only  provides  instruction 
for  using  the  keypad.  The  KF  does  not  instruct  partici¬ 
pants  on  the  use  of  the  numerical  keys  on  the  top  of  the 
keyboard. 


Analogy  Test 

The  Analogy  Test  measures  the  participant’s  reason¬ 
ing  ability  in  applying  the  correct  rules  to  solve  a  given 
problem.  The  participant  is  asked  to  determine  the 
relationship  of  the  words  or  pictures  in  set  A  and  use  this 
relationship  to  complete  an  analogy  in  set  B.  The 
following  paragraph  provides  observations  and  sugges¬ 
tions  for  this  test. 

Level  of  Difficulty 

The  vocabulary  level  and  the  types  of  relationships 
depicted  in  theAnalogyTest  may  have  been  too  difficult 
for  the  pilot  test  participants.  Perhaps  the  questions  can 
be  revised  to  require  a  lower  level  of  vocabulary  and 
reasoning  skills  for  the  participants. 

Classification  Test 

This  also  measures  the  participant’s  reasoning  ability 
in  applying  the  correct  rules  to  solve  a  given  problem. 
The  Classification  Test  is  similar  to  the  Analogy  Test, 
except  that  the  participant  is  required  to  determine  the 
relationship  of  three  words  or  pictures  and  use  this 
relationship  to  complete  the  series  with  a  fourth  word  or 
picture.  The  following  paragraph  provides  observations 
and  suggestions  for  the  improvement  of  this  test. 

Level  of  Difficulty 

Similar  to  the  issues  discussed  with  theAnalogyTest, 
many  of  the  items  in  the  Classification  T  est  appeared  to 
be  difficult  for  the  pilot  test  population.  The  Classifica¬ 
tion  Test  could  be  revised  to  allow  a  lower  level  of 
vocabulary  and  reasoning  skills. 

FEEDBACK  ON  TEST  BLOCK  E 

This  section  details  the  observations  and  suggestions 
for  the  five  tests  in  Block  E.  Specific  comments  about 
each  test  are  provided  below. 

Word  Memory  Test 

The  Word  Memory  Test  requires  the  participant  to 
remember  the  English  equivalents  for  words  in  an  artificial 
language  called  “Spron.”  The  following  paragraphs  pro¬ 
vide  observations  and  suggestions  for  this  test. 
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Level  of  Difficulty 

The  majority  of  participants  appeared  to  understand 
how  to  respond  to  this  test.  The  practice  session  for  this 
test  seemed  to  work  well  in  preparing  participants  for 
the  actual  test  questions. 

Erroneous  Text  Boxes 

Several  text  boxes  appear  during  this  test  that  should 
be  removed  for  future  versions  of  the  Word  Memory 
Test.  The  test  provides  the  participant  with  a  text  box  at 
the  end  of  the  test  that  displays  a  total  score.  This  is 
inconsistent  with  many  of  the  other  tests  in  the  AT -SAT 
Battery  that  provide  no  final  scores  to  the  participants. 
Also,  when  the  test  begins,  a  text  box  appears,  which 
prompts  the  participant  to  “Press  ‘Enter’  to  begin.” 
Once  the  participant  presses  enter,  another  text  box 
appears  that  prompts  the  participant  to  “Please  be  sure 
Num  Lock  is  engaged.”  Because  these  text  boxes  are 
irrelevant,  the  software  should  eliminate  this  message  in 
future  versions. 

Scan  Test 

The  Scan  Test  measures  a  participant’s  ability  to 
promptly  notice  relevant  information  that  is  continu¬ 
ously  moving  on  the  computer  screen.  Participants  are 
provided  with  a  number  range  and  asked  to  type  the 
identifier  for  numbers  that  appear  on  the  screen  outside 
of  that  range.  A  revised  version  of  this  test  was  installed 
midway  through  the  pilot  test,  which  changed  the 
process  for  recording  data  but  did  not  change  the 
appearance  or  the  performance  of  the  test  for  the  partici¬ 
pants.  The  following  paragraphs  provide  observations 
and  suggestions  for  the  improvement  of  this  test. 

Instructions 

While  the  instructions  for  the  test  seemed  clear, 
participants  had  some  common  misunderstandings  with 
the  test  instructions.  First,  participants  typed  the  actual 
numbers  which  were  outside  of  the  number  range 
instead  of  the  identifier  numbers.  This  confusion  might 
be  alleviated  by  revising  the  text  that  appears  on  the 
bottom  of  the  screen  during  the  test.  It  currently  states, 
“Type  the  identifier  numbers  contained  in  the  data 
blocks  with  the  lower  line  numbers  falling  beyond  the 
range.”  It  could  be  revised  to  state,  “Type  the  identifier 
numbers  contained  in  the  data  blocks  (following  the 
letter)  with  the  lower  line  numbers  falling  beyond  the 
range.”  Second,  participants  did  not  know  to  push 
“Enter”  after  typing  the  identification  numbers.  This 
confusion  might  be  alleviated  by  highlighting  the  text 


that  appears  at  the  bottom  of  the  screen  during  the  test 
to  “Press  ‘Enter’  to  record  this  selection.”  Third,  partici¬ 
pants  did  not  know  whether  the  instructions  to  identify 
numbers  “outside  the  range”  were  inclusive  of  the 
numbers  at  the  top  and  bottom  of  the  range.  This  issue 
should  be  explicitly  stated  in  the  test  instructions. 

Computer  Keyboards 

Since  the  directions  instructed  the  participants  to 
respond  as  quickly  as  possible,  in  their  haste,  many 
participants  were  pressing  the  numeric  keys  very  hard. 
The  banging  on  the  keyboard  was  much  louder  with  this 
test  than  with  any  of  the  other  tests;  this  affect  the 
longevity  of  the  numeric  keys  when  this  test  is  repeated 
numerous  times. 

Planes  Test 

The  Planes  Test  measures  the  participant’s  ability  to 
perform  different  tasks  at  the  same  time.  The  Planes 
Test  consists  of  three  parts.  In  Part  one,  the  participant 
uses  the  “  1  ”  and  the  “3”  keys  to  determine  whether  the 
red  plane  ( 1 )  or  the  white  plane  (3) ,  which  are  at  varying 
distances  from  their  destinations,  will  reach  its  destina¬ 
tion  first.  In  Part  two,  the  participant  uses  the  “1”  and 
the  “3”  keys  to  determine  if  a  statement  about  the  red 
and  white  planes  as  they  are  in  motion  is  true  (3)  or  false 
( 1 ).  In  Part  three,  the  participant  uses  the  “  1  ”  and  the  “3” 
keys  to  determine  if  a  statement  about  the  arrival  of  the 
red  and  white  planes  at  their  destination  are  true  (3)  or 
false  (1),  but  unlike  in  Part  two,  the  planes  are  at  varying 
distances  from  their  destinations.  The  following  para¬ 
graphs  provide  observations  and  suggestions  for  the 
improvement  of  this  test. 

Practice  Sessions 

The  practice  sessions  preceding  the  first  two  parts  of 
the  Planes  Tests  are  somewhat  lengthy.  There  are  24 
practice  items  that  the  participant  must  complete  before 
the  actual  test  of  96  items.  If  the  number  of  practice 
items  were  reduced  by  one  half,  the  participants  would 
still  have  enough  practice  without  becoming  bored 
before  the  actual  test  begins. 

Level  of  Difficulty 

Participants  appeared  to  be  challenged  by  the  Planes 
Test.  One  factor  that  added  to  the  level  of  difficulty  for 
the  participants  was  that  the  response  keys  for  Parts  two 
and  three  of  this  test  are:  1  =  “false”  and  3  =  “true.  ”  It  was 
more  intuitive  for  many  participants  that  1  =  “true”  and 
3  =  “false”  thus,  they  had  a  difficult  time  remembering 
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which  keys  to  use  for  true  and  false.  This  might  have 
caused  participants  more  difficulty  than  actually  deter¬ 
mining  the  correct  answer  to  the  statements.  If  the 
labeling  of  the  true  and  false  response  keys  cannot  he 
modified  in  future  software  versions,  a  message  hox  can 
be  created  to  remain  on  the  screen  at  all  times  that 
indicates  1  =  “false”  and  3  =  “true.” 

Test  Results 

Once  the  participant  provides  a  response  to  an  item 
on  the  Planes  Test,  a  results  screen  appears  indicating 
whether  the  response  was  “right”  or  “wrong.”  This  is 
inconsistent  with  many  of  the  other  tests  in  the  AT-SAT 
Battery  that  do  not  indicate  how  a  participant  performs 
on  individual  test  items,  in  addition  to  further  length¬ 
ening  an  already  lengthy  test. 

Angles  Test 

This  measures  a  participant’s  ability  to  recognize 
angles  and  perform  calculations  on  those  angles.  The 
following  paragraph  provides  observations  and  sugges¬ 
tions  for  this  test. 

Level  of  Difficulty 

Participants  appeared  to  be  challenged  by  this  test, 
although  it  seemed  as  if  they  could  either  very  quickly 
determine  a  response  about  the  measure  of  an  angle,  or 
it  took  them  some  time  to  determine  their  response. 

Applied  Mathematics  Test 

This  measures  the  participant’s  ability  to  apply  math¬ 
ematics  to  solve  problems  involving  the  traveling  speed, 
time,  and  distance  of  aircraft.  The  following  paragraphs 
provide  observations  and  suggestions  for  the  improve¬ 
ment  of  this  test. 

Instructions 

A  sentence  should  be  included  in  the  instructions 
that  no  pencils,  paper,  or  calculators  may  be  used  during 
this  test.  Many  pilot  test  participants  assumed  that  these 
instruments  were  allowed  for  this  portion  of  the  test. 

Level  of  Difficulty 

Many  participants  appeared  to  have  difficulty  deter¬ 
mining  the  best  answer  to  these  mathematical  ques¬ 
tions.  Several  participants  spent  so  much  time  trying  to 


calculate  an  answer  that  they  ran  out  of  time  and  were 
not  able  to  complete  this  test.  Perhaps  the  level  of 
difficulty  of  the  applied  mathematics  questions  can  be 
reduced. 

FEEDBACK  ON  THE  ENDING  BLOCK 

This  section  details  the  observations  and  suggestions 
for  the  four  retests  included  in  the  Ending  Block. 
Specific  comments  about  each  ending  test  block  is 
provided  below. 

Letter  Factory  Re-Test 

Participants  in  Group  One  (computers  1-10)  and 
Group  Five  (computers  41-50)  completed  a  re-test  of 
the  Letter  Factory  as  their  Ending  Block.  This  version  of 
the  Letter  Factory  Test  does  not  provide  the  participant 
with  any  test  instructions  or  opportunities  to  practice 
before  beginning  the  test.  However,  participants  ap¬ 
peared  to  have  little  difficulty  remembering  the  instruc¬ 
tions  for  this  test  from  Block  B. 

Air  Traffic  Scenarios  Re-Test 

Participants  in  Group  Two  during  the  pilot  test 
(computets  11-20)  completed  a  re- test  of  the  Air  T  raffic 
Scenarios  as  their  Ending  Block.  This  re-test  allows  the 
participant  to  review  the  instructions  before  beginning 
the  abbreviated-length  version  of  the  Air  Traffic  Sce¬ 
narios.  The  proposed  revisions  to  the  Air  Traffic  Sce¬ 
narios  Test  in  Section  4  of  this  report  also  apply  to  this 
version  of  the  test  in  the  Ending  Block. 

Static  Vector/ Continuous  Memory  and  Word 
Memory  Re-Test 

Participants  in  Group  Three  (computers  2 1  -30)  com¬ 
pleted  a  re-test  of  the  Static  Vector/ Continuous  Memory 
Test  and  the  Word  Memory  Test  as  their  Ending  Block. 
The  re-test  of  the  Static  Vector/Continuous  Memory 
Test  allows  the  participant  to  review  the  instructions 
but  does  not  provide  a  practice  session  before  the  actual 
test  begins.  The  proposed  revisions  to  the  Static  Vector/ 
Continuous  Memory  Test  in  Section  6.2  of  this  report 
and  to  the  Word  Memory  Test  in  Section  8.1  of  this 
report  also  apply  to  these  versions  of  the  tests  in  the 
Ending  Block. 
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Word  Memory  and  Time  Wall/Pattern  Recognition 
Re-Test 

Participants  in  Group  Four  (computers  31-40)  com¬ 
pleted  a  re-test  of  Word  Memory  and  the  Time  Wall/ 
Pattern  Recognition  as  their  Ending  Block.  The  re-test 
of  the  Time  Wall/Pattern  Recognition  Re-Test  allows 
the  participant  to  review  the  instructions  and  complete 
a  practice  session  before  beginning  the  test.  The  pro¬ 
posed  revisions  to  the  Word  Memory  Test  in  Section 
8. 1  of  this  report  and  the  Time  Wall/Pattern  Recogni¬ 
tion  Test  in  Section  7. 1  of  this  report  also  apply  to  these 
versions  of  the  tests  in  the  Ending  Block. 

SUMMARY  OF  THE  FEEDBACK  ON  THE 
AT-SAT  PILOT  TEST  BATTERY 

This  section  of  the  report  summarizes  the  feedback 
on  all  the  test  blocks  within  the  AT-SAT  Pilot  Test 
Battery.  Overall,  we  are  recommending  relatively  few 


changes  to  the  entire  battery  of  tests.  The  maj  ority  of  the 
recommended  changes  are  intended  to  enhance  the 
clarity  of  test  instructions,  increase  the  value  of  the  test 
practice  sessions,  and  revise  some  of  the  questions  for 
the  ability  level  of  the  participants.  Exhibit  3.2.3,  on  the 
following  page,  displays  a  summary  of  the  proposed 
revisions  to  the  pilot  test  software. 

The  information  provided  by  the  Test  Administra¬ 
tors  was  one  of  the  information  sources  used  to  revise  the 
test  battery.  A  significant  effort  on  the  part  of  the  project 
team  went  into  revising  the  instructions  for  the  tests  and 
the  other  changes  recommended  by  the  Test  Adminis¬ 
trators.  The  next  section  discusses  the  psychometric 
information  used  to  revise  the  battery.  Both  sources  of 
information  provided  the  test  developers  the  informa¬ 
tion  necessary  to  build  the  Beta  Battery,  which  was  used 
in  the  concurrent  validation  study. 
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This  chapter  outlines  the  rationale  used  in  revising 
the  tests  and  is  based  on  the  pilot  test  data  gathered  prior 
to  the  validation  study.  A  full  description  of  the  samples 
used  in  the  pilot  study  can  be  found  in  Chapter  3.2.  It 
is  important  to  note  that  some  of  the  tests  were  devel¬ 
oped  specifically  for  use  in  the  AT-SAT  validation 
study,  and  therefore  it  was  imperative  that  they  be  pilot- 
tested  for  length,  difficulty,  and  clarity.  There  were  two 
levels  of  analysis  performed  on  the  pilot  test  data.  First, 
logic  and  rationale  were  developed  for  the  elimination 
of  data  from  further  consideration  in  the  analyses.  After 
the  were  elimination  process,  an  item  analysis  of  each 
test  was  used  to  determine  the  revisions  to  tests  and 
items  that  were  needed. 

Exclusionary  decision  rules  were  based  on  available 
information,  which  varied  from  test  to  test.  For  ex¬ 
ample,  in  some  instances,  item  latency  (time)  informa¬ 
tion  was  available  as  the  appropriate  method  for 
exclusion;  in  other  cases,  the  timing  of  the  tests  were 
computer  driven  and  other  criteria  for  exclusion  were 
developed.  An  item  was  considered  a  candidate  for  dele¬ 
tion  if  it  exhibited  any  of  the  following  characteristics: 

•  Low  Discrimination:  The  item  did  not  discriminate 
between  those  individuals  who  received  high  versus  low 
total  scores,  stated  as  a  biserial  correlation. 

•  Check  Option:  One  or  more  incorrect  response  op¬ 
tions  had  positive  biserial  correlations  with  total  test 
score. 

•  Too  Hard:  The  percent  correct  was  low. 

•  Too  Easy:  The  percent  correct  was  high. 

•  High  Omits:  The  item  was  skipped  or  not  reached, 
with  these  two  problems  being  distinguishable  from 
each  other. 

Applied  Math  Test 

Case  Elimination 

To  determine  reasonable  average  and  total  latencies 
for  the  items  attempted,  the  original  sample  of 435  was 
restricted  to  those  individuals  who  completed  all  53 


items  (N=392)  ofthe  Applied  Math  test  (AM).  Examining 
the  average  latency  in  seconds  for  the  items  revealed  a  mean 
time  of  1 4.7  and  a  standard  deviation  of  1 0.  After  review  of 
the  aaual  test  items,  it  was  decided  that  any  individual 
spending  less  than  4  per  item  was  probably  responding 
randomly  or  inappropriately.  Review  of  a  scatter  plot  of 
average  latency  by  percentage  correct  revealed  that  those 
individuals  taking  less  than  5  scored  at  the  extreme  low  end, 
about  half  scoring  below  chance.  To  corroborate  this 
information,  a  comparison  of  scores  on  the  Applied  Math 
test  and  scores  on  the  ASVAB  Arithmetic  Reasoning  test 
(AS_AR)  identified  individuals  who  had  the  mathematical 
ability  but  were  not  motivated  to  perform  on  the  Applied 
Math  test  (i.e.,  high  ASVAB  score  but  low  AM  score). 

Based  on  this  information,  three  guidelines  for  elimi¬ 
nating  individuals  were  formulated: 

( 1 )  High  Omits:  It  was  determined  that  any  individual 
attempting  fewer  than  35  items  AND  receiving  a  per¬ 
cent  correct  score  of  less  than  39  percent  was  not 
making  a  valid  effort  on  this  test. 

(2)  Random  Responders:  After  reviewing  and  compar¬ 
ing  the  percent  correct  scores  for  the  Applied  Math  test 
and  the  AS_AR  scores,  it  was  determined  that  any 
individual  whose  AS_AR  was  greater  than  63,  but 
whose  percent  correct  was  less  than  23%,  was  not 
putting  forth  an  honest  effort  on  this  test. 

(3)  Individuals  whose  average  latency  was  less  than  4 
were  excluded  from  further  item  analysis. 

Application  of  these  exclusion  rules  further  reduced 
the  sample  size  to  358  for  the  item  analysis. 

Item  Analysis 

On  the  Applied  Math  test,  all  of  the  items  that 
were  characterized  as  High  Omits  were  items  that  the 
participants  did  not  reach  because  of  test  length,  not 
items  that  they  merely  skipped.  Additionally,  this  test 
has  four  response  options  with  each  item,  and  there¬ 
fore  the  chance  level  of  a  correct  response  is  25%. 
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After  review  of  the  item  analysis,  18  items  were 
deleted,  reducing  the  test  length  to  30  items.  The  item 
analysis  printout  for  the  deleted  items  can  he  found  in 
Appendix  A.  An  extensive  review  of  the  items  by  content 
and  computation  type  was  conducted  to  ensure  fiiir 
representation  of  relevant  item  types.  The  item  types 
represented  were  Computing  Distances,  Computing 
Travel  Time,  Computation  Given  Multiple  Point  Dis¬ 
tances,  Computing  Ascending/Descending  Rates,  and 
Computing  Speed. 

Summary  and  Recommendations 

The  test  was  shortened  from  53  items  to  30.  Textual 
changes  were  made  to  four  items  for  clarification.  The 
items  were  re-ordered  with  the  five  easiest  items  first, 
then  the  rest  of  the  30  items  randomly  distributed 
throughout  the  test.  This  ensured  that  the  test  taker 
would  reach  at  least  some  of  the  most  difficult  items. 

Dials  Test 

Case  Elimination 

For  the  Dials  test,  406  of  the  449  participants  com¬ 
pleted  the  entire  test.  A  scatter  plot  of  average  latency  per 
item  in  seconds  by  percent  correct  for  attempted  items 
was  created  for  the  reduced  sample  (406).  The  mean  and 
standard  deviation  for  average  latency  were  12.47  and 
4.11,  respectively.  The  mean  and  standard  deviation  for 
percentage  correct  were  78.89%  and  12.96%,  respec¬ 
tively.  A  grid  overlay  based  on  the  means  and  standard 
deviations  revealed  that  individuals  who  were  more 
than  two  standard  deviations  below  the  mean  for  aver¬ 
age  latency  (4.25  per  item)  were  scoring  more  than  two 
standard  deviations  below  the  mean  for  percent  correct 
(52.97%).  It  appears  that  these  individuals  were  not 
taking  the  time  to  read  the  items  or  put  forth  their  best 
effort.  Following  an  exclusion  rule  of  eliminating  par¬ 
ticipants  who  had  an  average  latency  per  item  of  4.25  or 
less,  the  sample  was  reduced  from  449  to  44 1 . 

Item  Analysis 

After  review  of  the  item  analysis  and  of  specific 
items,  13  items  were  deleted  from  the  original  test. 
All  had  low  discrimination  and/or  another  response 
option  that  was  chosen  more  frequently  than  the 
correct  response.  In  many  instances,  the  graphics 
made  it  difficult  to  discriminate  between  correct  and 
incorrect  dial  readings.  The  revised  test  consists  of  44 
items.  The  item  analysis  printout  for  the  deleted 
items  can  be  found  in  Appendix  A. 


Summary  and  Recommendations 
The  1 3  items  that  had  low  discrimination  or  response 
options  that  were  chosen  more  often  than  the  correct 
response  were  eliminated,  reducing  the  test  length  to  44 
items.  An  additional  recommendation  was  that  17  - 
inch  display  monitors  be  used  in  the  beta  vetsion  to 
ensure  the  integrity  of  the  graphics. 

Angles  Test 

Case  Elimination 

For  the  Angles  test,  all  445  individuals  completed  the 
entire  test  (30  items).  A  scatter  plot  was  created  of  the 
average  latency  per  item  in  seconds  by  the  percent 
correct  for  attempted  items.  The  mean  and  standard 
deviation  for  average  latency  wete  8.2  and  2.67,  respec¬ 
tively.  The  mean  and  standard  deviation  for  percent 
correct  were  67.73%  and  17.7%,  respectively.  A  grid 
based  on  the  means  and  standard  deviations  of  each  axis 
revealed  that,  of  the  four  individuals  who  were  more 
than  two  standard  deviations  below  the  mean  for  aver¬ 
age  latency  (2.86  per  item),  three  scored  more  than  two 
standard  deviations  below  the  mean  for  percentage 
correct  (32.33%).  The  other  individual  was  about  1.5 
standard  deviations  below  the  mean  for  percent  correct. 
It  appears  that  these  individuals  were  not  taking  the  time 
to  read  the  items  or  put  forth  their  best  effort.  By 
eliminating  those  individuals  with  an  average  item 
latency  of  less  than  2.86,  the  item  analysis  sample  was 
reduced  to  44 1 . 

Item  Analysis 

The  item  analysis  did  not  reveal  any  problem  items 
and  there  appeared  to  be  a  good  distribution  of  item 
difficulties.  No  text  changes  were  indicated.  After  re¬ 
viewing  the  item  analysis  and  the  items  in  the  test,  none 
of  the  items  were  deleted. 

Summary  and  Recommendations 

This  test  appears  to  function  as  it  was  intended. 
There  were  no  item  deletions  and  no  textual  changes. 

Sound  Test 

Case  Elimination 

On  the  Sound  test,  437  participants  completed  1 7  or 
18  items.  Of  the  remaining  five  participants,  one  com¬ 
pleted  only  two  items  (got  none  correct)  and  was  deleted 
from  the  sample.  The  other  four  participants  made  it  to 
the  fourth  set  of  numbers  (8  digits).  All  the  scores  of  this 
group  of  four  were  within  one  standatd  deviation  (15%) 
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of  the  mean  of  the  percentage  correct  for  attempted 
items  (35.6%).  Additionally,  five  other  participants  did 
not  get  any  items  correct.  It  was  determined  that  two  of 
them  were  not  trying,  and  they  were  deleted.  The 
remaining  three  seemed  to  “be  in  the  ballpark”  with 
their  responses  (i.  e. ,  many  of  their  responses  were  almost 
correct).  With  the  exclusion  of  three  participants,  the 
total  sample  for  the  item  analysis  was  439. 

Alternative  Scoring  Procedure 

The  Sound  test  consists  of  numbers  of  set  lengths  (5, 
6,  7,  8,  9,  10  digits)  being  read  and  then  participants 
recalling  them.  There  are  three  trials  associated  with 
each  number  length  (i.e.,  number  length  5  has  three 
trials,  number  length  6  has  three  trials,  etc.)  for  a  total 
of  1 8  items.  Examinees  receive  a  point  for  every  item 
answered  correctly.  An  alternative  scoring  procedure 
would  be  to  award  a  point  for  each  digit  they  get  correct 
and  one  point  for  a  digit  reversal  error.  For  example,  in 
the  5-digit  case,  a  correct  response  may  be  12345,  but  a 
participant  may  answer  12354  (digit  reversal).  In  this 
case,  the  participant  would  receive  3  points  for  the  first 
three  digits  and  1  point  for  the  digit  reversal,  for  a  total 
of  4  points  on  that  trial.  This  scoring  procedure  was 
examined  as  an  alternative  to  the  number  correct  score. 

Item  Analysis 

After  review  of  the  item  analysis,  none  of  the  items 
were  removed.  However,  the  biserial  correlations  of  the 
items  from  digit  length  5  and  digit  length  10  were 
appreciably  lower  than  the  rest  of  the  items.  The  reliabil¬ 
ity  of  this  test  with  the  original  scoring  procedure  was 
.70,  while  the  alternative  scoring  procedure  improved 
reliability  to  .77.  Using  the  alternative  scoring  proce¬ 
dure,  in  a  comparison  of  the  original  version  and  a 
revised  version  with  digit  length  5  and  digit  length  10 
removed,  the  revised  version  had  a  slightly  higher  reli¬ 
ability  (.78). 

Summary  and  Recommendations 

Since  digit  lengths  of  5  and  10  had  lower  biserial 
correlations  than  the  rest  of  the  items,  it  was  recom¬ 
mended  that  the  number  of  trials  associated  with  these 
items  be  reduced  to  two  each.  The  alternative  scoring 
procedure,  based  on  the  number  of  within-item  digits 
correct  with  partial  credit  for  digit  reversals,  was  recom¬ 
mended  for  the  beta  version. 


Memory  Test 

Case  Elimination 

A  scatter  plot  of  Memory  test  items  answered  by 
percent  correct  revealed  a  sharp  decline  in  the  percent 
correct  when  participants  answered  fewer  than  1 4  items. 
It  was  decided  that  participants  who  answered  fewer 
than  14  items  were  not  making  an  honest  effort  on  this 
test.  Additionally,  it  was  felt  that  participants  who 
scored  less  than  5%  correct  (about  1  of  24  correct) 
probably  did  not  put  forth  their  best  effort,  and  there¬ 
fore,  they  were  removed  from  the  item  analyses.  These 
two  criteria  eliminated  1 4  participants,  leaving  a  sample 
of  435  for  the  item  analyses. 

Item  Analysis 

After  review  of  the  item  analysis,  none  of  the  items 
were  removed.  Item  1  had  low  discrimination,  low 
percent  correct,  and  a  high  number  of  omits.  However, 
there  were  no  such  problems  with  the  remaining  items, 
and  given  that  these  are  non-sense  syllables,  one  expla¬ 
nation  may  attribute  the  poor  results  to  first-item  ner¬ 
vousness-acclimation.  All  items  were  retained  for  the 
beta  version,  and  no  editorial  changes  were  made. 

Summary  and  Recommendations 

This  test  performed  as  expected  and  had  a  normal 
distribution  of  scores.  One  item  had  problem  character¬ 
istics,  but  a  likely  explanation  may  be  that  it  was  the  first 
item  on  the  test.  The  recommendation  was  to  leave  all 
24  items  as  they  were  but  to  re-examine  the  suspect  item 
after  beta  testing.  If  the  beta  test  revealed  a  similar 
pattern,  then  the  item  should  be  examined  more  closely. 

Analogy  Test 

Case  Elimination 

For  the  Analogy  test,  cases  were  eliminated  based  on 
three  criteria:  missing  data,  pattern  responding,  and 
apparent  lack  of  participant  motivation. 

Missing  Data.  The  test  software  did  not  permit 
participants  to  skip  items  in  this  test,  but  several  (12.8%) 
did  not  complete  the  test  in  the  allotted  time,  resulting 
in  missing  data  for  these  cases.  Those  missing  20%  or 
more  of  the  data  (i.e.,  cases  missing  data  for  1 1  items  or 
more)  were  omitted.  Five  cases  were  eliminated  from  the 
sample. 

Pattern  Responders:  The  chance  level  of  responding 
for  this  test  was  20%.  An  examination  of  those  partici¬ 
pants  near  chance  performance  revealed  one  case  where 
the  responses  appeared  to  be  patterned  or  inappropriate. 
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Unmotivated  Participants:  Identifying  participants 
who  appeared  to  be  unmotivated  was  based  on  the 
average  latency  per  item,  which  was  5.4.  It  was  deter¬ 
mined,  because  of  the  complexity  of  the  items,  that 
participants  spending  5.4  or  less  were  not  taking  the  test 
seriously  or  were  randomly  responding,  and  therefore 
were  eliminated  from  the  item  analyses.  As  a  cross  check, 
an  examination  of  the  percentage  correct  for  those 
participants  whose  average  latency  was  5.4  seconds  or 
less  showed  that  their  scores  were  near  chance  levels. 
Four  participants  were  eliminated. 

In  summary,  10  participants  were  eliminated  from 
further  analyses,  reducing  the  sample  size  from  449  to 
439. 

Scale  and  Item  Analyses 

An  examination  of  the  biserial  correlations  for  the  53 
items  revealed  12  items  that  had  biserial  correlations  of 
.10  or  less.  This  reduced  the  number  of  items  within 
three  of  the  four  test  scales  as  follows  (the  original 
number  of  items  appears  in  parentheses):  Non-Seman- 
ticWords9  (15),  Semantic  Words  12  (15),  and  Seman¬ 
tic  Visuals  7  (10).  Tables  3.3.1  to  3.3.4  present  these 
corrected  item-total  correlations  and  the  alphas  for  the 
items  within  each  scale.  As  Table  3.3.4  indicates,  all  13 
items  within  the  Non-Semantic  Visual  scale  met  the 
criterion  for  retention.  After  omitting  items  based  on 
the  above  criteria,  the  number  of  items  in  this  test 
dropped  from  53  to  4 1 . 

Construct  Validity 

A  multitrait-multimethod  matrix  was  constructed 
to  assess  whether  the  information  processing  scores  and 
the  number-correct  scores  measure  different  constructs 
or  traits.  Test  scores  based  on  two  traits  (i.e..  Informa¬ 
tion  Processing  and  Reasoning)  and  four  methods  (i.e.. 
Word  Semantic,  Word  Non-Semantic,  Visual  Seman¬ 
tic,  and  Visual  Non-Semantic)  were  examined.  The 
results  provided  the  following  median  correlations: 

•  The  median  convergent  validity  (i.e.,  same  trait,  different 
method)  for  information  processing  scores  was  .49. 

•  The  median  convergent  validity  for  number-correct  scores 
was  .34, 

•  The  median  divergent  validity  (i.e.,  different  trait,  differ¬ 
ent  method)  was  .18. 

These  preliminary  results  suggest  keeping  separate 
the  information-processing  and  number-correct  scores 
for  the  Analogy  test,  pending  further  investigation. 


Testing  Time 

Based  on  the  sample  of 439  participants,  95%  of  the 
participants  completed  the  test  and  instructions  in  33 
minutes  (Table  3.3. 5) .  Table  3.3.6  shows  time  estimates 
for  two  different  levels  of  reliability. 

Test  Revisions 

A  content  analysis  of  the  test  revealed  four  possible 
combinations  of  semantic/non-semantic  and  word/vi¬ 
sual  item  types.  The  types  of  relationships  between  the 
word  items  could  be  (a)  semantic  (word-semantic),  (b) 
based  on  a  combination  of  specific  letters  (word  -  non- 
semantic),  (c)  phonetic  (word  -  non-semantic),  and  (d) 
based  on  the  number  of  syllables  (word  -  non-semantic). 
The  types  of  relationships  between  visual  items  could  be 
based  on  (a)  object  behavior  (visual-semantic),  (b)  ob¬ 
ject  function  (visual-semantic),  (c)  object  feature  (vi¬ 
sual-semantic,  (d)  adding/deleting  parts  of  the  figures 
(visual  -  non-semantic),  (e)  moving  parts  of  the  figures 
(visual  -  non-semantic),  and  (f)  rotating  the  figures 
(visual  -  non-semantic). 

After  categorizing  the  items  based  on  item  type,  an 
examination  of  the  item  difficulty  level,  item-total 
correlations,  the  zero-order  intercorrelations  between 
all  items,  and  the  actual  item  content  revealed  only  one 
perceptible  pattern.  Six  non-semantic  word  items  were 
removed  due  to  low  item-total  correlations,  five  being 
syllable  items  (i.e.,  the  correct  solution  to  the  analogy 
was  based  on  number  of  syllables). 

Seven  more  items  were  removed  from  the  alpha 
Analogy  test  version  due  to  either  very  high  or  low 
difficulty  level,  or  to  having  poor  distractor  items. 

Word  Items.  The  time  allocated  to  the  Analogy  test 
items  remained  approximately  the  same  (35  minutes 
and  10  minutes  for  reading  instructions)  from  the  alpha 
version  to  the  beta  version.  The  number  of  word  items 
did  not  increase;  however,  nine  items  were  replaced  with 
items  that  had  similar  characteristics  of  other  well¬ 
performing  word  items.  There  were  equal  numbers  of 
semantic  and  non-semantic  items  (15  items  each). 

Since  the  analogy  items  based  on  the  number  of 
syllables  performed  poorly,  this  type  of  item  was  not 
used  when  replacing  the  non-semantic  word  items. 
Instead,  the  five  non-semantic  word  items  were  replaced 
with  combinations  of  specific  letters  and  phonetic  items. 
Additionally,  three  semantic  items  were  replaced  with 
three  new  semantic  items  of  more  reasonable  (expected) 
difficulty  levels. 
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Visual  Items.  Since  the  non-semantic  picture  items 
demonstrated  a  relatively  stable  alpha  (.67)  and  high 
item-total  correlations,  no  items  were  removed.  In  an 
effort  to  stabilize  the  alpha  further,  three  non-semantic 
picture  items  were  added,  increasing  the  non-semantic 
visual  subtest  from  13  to  16  items. 

One  item  was  dropped  because  it  contained  poor 
distractors.  Two  other  semantic  visual  items  that  ap¬ 
peared  to  have  poor  distractors  were  modified  to  im¬ 
prove  the  clarity  of  the  items  (without  lowering  the 
difficulty  level).  In  addition,  two  newly  created  items 
were  added  to  this  scale.  Thus,  one  item  was  replaced 
with  two  new  items,  and  two  others  were  modified. 

Instructional  Changes.  Based  on  feedback  from  site 
Test  Administrators,  portions  of  the  Analogy  test  in¬ 
structions  were  simplified  to  reduce  the  required  read¬ 
ing  level  of  the  text.  Also,  the  response  mode  was 
changed  from  use  of  a  keyboard  to  use  of  a  mouse.  The 
Viewing  an  Item  section  of  the  instructions  was  revised 
accordingly. 

Summary  and  Recommendations 
The  Analogy  test  assesses  inductive  reasoning  and 
information  processing  abilities  in  four  areas:  Non- 
Semantic  Word,  Semantic  Word,  Non-Semantic  Vi¬ 
sual,  and  Semantic  Visual.  The  number-correct  scores 
that  reflected  reasoning  ability  proved  less  reliable  than 
the  information  processing  scores.  Of  the  53  items  in 
the  alpha  battery,  4l  contributed  sufficiently  to  test 
reliability  to  warrant  inclusion  in  the  revised  version.  It 
was  estimated  that  to  achieve  a  reliability  level  of  0.80  it 
would  be  necessary  to  increase  the  test  length  to  150 
items.  Given  time  limits  in  the  validation  version,  the 
overall  test  length  was  limited  to  57  items. 

Classification  Test 

Case  Elimination  Analyses 
Cases  in  the  Classification  test  were  eliminated  based 
on  three  criteria:  missing  data,  pattern  responding,  and 
apparent  lack  of  participant  motivation. 

Missing  Data.  As  with  the  Analogy  test,  the  Classi¬ 
fication  test  software  did  not  permit  participants  to  skip 
items.  However,  some  participants  (7.5%)  did  not 
complete  the  test  in  the  allotted  time,  resulting  in 
missing  data.  Of  these  cases,  those  missing  20%  or  more 
of  the  data  (i.e.,  cases  missing  data  for  nine  items  or 
more)  were  omitted.  A  total  of  1 0  cases  were  eliminated. 


Pattern  Responding.  From  examination  of  the  pat¬ 
tern  of  responses  of  participants  who  scored  at  or  near 
chance  levels  (20%),  eight  participants  were  identified 
as  responding  randomly  and  were  eliminated. 

Unmotivated  Participants.  It  was  decided  that  par¬ 
ticipants  spending  less  than  3  per  item  were  not  making 
a  serious  effort.  Four  participants  fell  into  this  category. 
An  examination  of  their  total  scores  revealed  that  they 
scored  at  or  near  chance  levels,  and  thus  they  were 
eliminated  from  further  analyses. 

In  summary,  22  participants  were  eliminated  from 
further  analyses,  reducing  the  sample  size  for  this  test 
from  449  to  427. 

Scale  Reliabilities  and  Item  Analyses 

Reliability  analyses  were  conducted  to  identify  the  items 
within  each  of  the  four  test  scales  that  did  not  contribute  to 
the  internal  consistency  of  that  scale.  The  corrected  item- 
total  correlation  was  computed  for  each  item  within  a 
scale,  as  well  as  the  overall  alpha  for  that  scale. 

An  examination  of  the  item-total  correlations  re¬ 
vealed  that  the  Non-Semantic  Word  scale  items  had  an 
average  correlation  of .  1 79,  and  therefore  the  entire  scale 
was  omitted  from  further  analyses.  This  reduced  the 
number  of  items  within  the  three  remaining  test  scales 
as  follows  (the  original  number  of  items  appears  in 
parentheses):  Semantic  Word  9  (11),  Non-Semantic 
Visual  10(13),  and  Semantic  Visual  3(10).  Note  that 
the  greatest  number  of  items  were  removed  from  the 
semantic  visual  scale.  Tables  3.3.7  to  3.3. 1 0  present  the 
corrected  item-total  correlations  for  the  items  within 
each  scale.  After  omitting  items  based  on  the  above 
criteria,  the  number  of  items  in  this  test  was  reduced 
from  46  to  22. 

Construct  Validity 

In  assessing  the  construct  validity  of  the  information 
processing  measures  independent  of  the  number  correct 
scores,  a  multitrait-multimethod  matrix  was  constructed. 
Two  traits  (i.e.,  information  processing  and  reasoning) 
and  four  methods  (i.e..  Word  Semantic,  Word  Non- 
Semantic,  Visual  Semantic,  and  Visual  Non-Semantic) 
were  examined.  The  results  of  this  analysis  provided  the 
following  median  correlations: 

•  The  median  convergent  validity  (i.e.,  same  trait,  different 
method)  for  information  processing  scores  was  .48. 

•  The  median  convergent  validity  for  number-correct 
scores  was  .20. 
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•  The  median  divergent  validity  (i.e.,  different  trait, 
different  method)  was  .09. 

These  preliminary  results  suggested  a  separation  of 
the  information-processing  and  number-correct  scores 
for  the  Classification  test,  pending  further  investiga¬ 
tion. 

Time  Limit  Analyses 

Based  on  the  sample  of 427  participants,  95%  of  the 
participants  completed  the  instructions  and  the  46  test 
items  in  22  minutes  (Tables  3.3.11).  Table  3.3.12 
shows  estimates  of  test  time  limits  assuming  two  differ¬ 
ent  levels  of  reliability  and  test  lengths  for  the  three  test 
parts.  These  estimates  assume  keeping  all  aspects  of  the 
test  the  same  (i.e.,  all  four  classification  schemes). 

Summary  and  Recommendations 

Of  the  original  46  items,  only  three  of  the  four  scales 
(i.e..  Semantic  Word,  Semantic  Visual,  and  Non-Se- 
mantic  Visual)  and  a  total  of  22  items  contributed 
sufficiently  to  test  reliability  to  warrant  inclusion  in  a 
revised  test  version.  To  construct  a  test  having  the  same 
three  parts  and  increase  the  reliability  to  about  .80  (for 
number-correct  scores),  the  number  of  items  would 
need  to  increase  from  the  22  to  1 39.  It  was  further  found 
that  the  Classification  test  correlates  highly  with  the 
Analogy  test.  Given  that  the  Classification  test  had 
lower  reliability  scores  than  the  Analogy  test,  it  was 
recommended  that  the  Classification  test  be  eliminated 
from  the  AT-SAT  battery. 

Letter  Factory  Test 

Analysis  of  Initial  LFT 

Case  Elimination.  Two  criteria  were  used  in  elimi¬ 
nating  Letter  Factory  Test  participants:  apparent  lack  of 
participant  motivation  and  inappropriate  responding. 

Unmotivated  Participants.  Unmotivated  participants 
were  considered  to  be  those  who  responded  to  very  few 
or  none  of  the  items.  An  examination  of  performance  on 
the  number  correct  across  all  Planning/Thinking  (P/T) 
items  in  the  test  sequences  (Table  3.3.13)  reveals  a  gap 
in  the  distribution  at  28%  correct.  It  was  decided  that 
participants  scoring  below  28%  were  not  making  a 
serious  effort  on  this  test,  and  they  were  eliminated  from 
further  analysis. 

Inappropriate  Responding.  Inappropriate  respond¬ 
ers  were  identified  as  participants  who  either  selected  a 
box  of  tbe  wrong  color  or  selected  boxes  when  none  were 


needed.  During  the  test,  there  were  86  times  when  a 
participant  should  have  placed  a  box  in  tbe  loading  area. 
The  computer  software  recorded  the  number  of  times  a 
participant  tried  to  place  a  box  incorrectly  (i.e.,  to  place 
a  box  when  one  was  not  needed  or  to  place  an  incorrectly 
colored  box).  This  measure  serves  as  an  indicator  of 
inappropriate  responding.  Table  3.3.14  shows  the  dis¬ 
tribution  of  the  number  of  unnecessary  attempts  to 
place  a  box  in  the  loading  area  across  the  entire  sample 
of  44 1  cases. 

Several  participants  had  a  very  high  number  of  these 
erroneous  mouse  clicks.  There  are  two  possible  reasons 
for  this.  Feedback  from  the  Test  Administrators  indi¬ 
cated  that  some  participants  were  double-clicking  the 
mouse  button,  instead  of  clicking  once,  in  order  to 
perform  LFT  test  tasks.  Every  instance  a  participant 
erroneously  clicks  the  mouse  button  is  recorded  and 
compiled  by  the  computer  to  generate  an  inappropriate 
response  score.  Thus,  if  a  participant  orders  the  correct 
number  of  boxes  (86)  by  double-clicking  instead  of 
single-clicking,  86  points  will  be  added  to  his  or  her 
inappropriate  response  score. 

A  few  participants  had  a  random-response  score  higher 
than  86.  These  participants  may  have  developed  a  strategy 
to  increase  their  test  score.  The  test  instructions  explained 
the  importance  ofplacing  a  box  in  the  loading  area  as  soon 
as  one  was  needed.  This  may  have  caused  some  partici¬ 
pants  to  continuously  and  consistently  attempt  to  place 
boxes  in  the  loading  area.  Participants  received  an  error 
signal  each  time  they  unnecessarily  attempted  to  place  a 
box;  however,  they  may  not  have  realized  the  negative 
impact  of  this  error  on  their  test  score. 

Cases  were  eliminated  where  the  inappropriate  re¬ 
sponse  score  was  higher  than  86.  This  allowed  using  the 
data  from  participants  who  were  motivated  but  might 
have  misunderstood  the  proper  way  to  use  a  mouse 
during  this  test.  However,  to  prevent  an  inappropriate 
response  strategy  from  interfering  with  a  true  measure  of 
Planning/Thinking  Ahead  (P/T),  informarion  from  the 
inappropriate  response  variable  must  be  used  when 
calculating  a  participant’s  P/T  test  score.  Omitting  cases 
based  on  the  above  criteria  reduced  tbe  sample  from  44 1 
to  405. 

Item  Analyses 

Recall  From  Interruption  (RI).  The  proposed  mea¬ 
sure  of  R1  was  a  difference  score  between  a  participant’s 
number-correct  score  across  a  set  of  items  presented 
immediately  after  an  interruption  and  number-correct 
score  across  a  set  of  items  presented  just  before  the 
interruption.  Four  of  the  test  sequences  (sequences  4, 6, 
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8,  and  1 1)  contained  RI  items.  Table  3.3.15  shows  the 
number  of  items  within  each  sequence  that  make  up 
each  pre-interruption  and  post-interruption  scale  score, 
as  well  as  the  score  means  and  standard  deviations.  The 
mean  scores  for  each  sequence  are  very  high,  indicating 
a  ceiling  effect.  However,  increasing  the  difficulty  of  the 
RI  items  would  require  increasing  either  the  number  of 
letters  on  belts  or  the  belt  speed.  Either  of  these  methods 
would  alter  the  task  so  that  psychomotor  ability  would 
become  a  very  significant  component  of  the  task.  There¬ 
fore,  it  was  concluded  that  this  task  is  not  suited  to 
providing  a  measure  of  RI. 

Table  3.3. 1 5  also  provides  a  summary  of  the  reliabil¬ 
ity  of  the  pre-interruption,  post-interruption,  and  dif¬ 
ference  scores.  Notice  that  the  reliability  of  the  pre-  and 
post-interruption  scores  (Alpha  =  .79  and  .73,  respec¬ 
tively)  is  much  higher  than  the  reliability  of  the  differ¬ 
ence  scores  (v^lpha  =  .10).  Plans  for  the  recall  from 
intermption  score  were  abandoned  due  to  low  reliability. 

Planning/Thinking  Ahead.  To  prevent  an  inappro¬ 
priate  response  strategy  from  interfering  with  a  true 
measure  of  P/T,  the  inappropriate  response  score  must 
be  used  in  calculating  a  participant’s  P/T  test  score.  The 
test  design  prevented  the  association  of  unnecessary 
mouse  clicking  with  a  specific  P/T  item.  (Participants 
do  not  have  to  respond  to  P/T  test  items  in  the  order  in 
which  they  receive  them;  instead,  they  may  wait  and 
then  make  several  consecutive  mouse  clicks  to  place 
multiple  boxes  in  the  loading  area.)  However,  the 
software  recorded  the  numbet  of  times  each  participant 
inappropriately  attempted  to  place  a  box  during  a  test 
sequence.  Also,  the  number  of  unnecessary  mouse  clicks 
has  a  low  but  significant  negative  correlation  (r  =  -.20, 
p  <  .001)  with  the  number-correct  scores  on  a  sequence. 
Therefore,  the  P/T  scale  score  per  sequence  was  com¬ 
puted  by  subtracting  the  number  of  unnecessary  mouse 
clicks  in  a  sequence  from  the  number-correct  score 
across  all  P/T  items  in  that  sequence. 

Table  3.3.16  summarizes  findings  from  the  reliabil¬ 
ity  analysis  on  the  seven  sequences  designed  to  measure 
P/T.  The  second  column  indicates  the  number  of  P/T 
items  involved  in  calculating  the  P/T  sequence  scores. 
Notice  that  the  sequence-total  correlations  are  .60  or 
higher.  Therefore,  none  of  the  P/T  sequences  were 
deleted.  The  alpha  computed  on  the  seven  sequences 
was  .86. 

Situational  Awareness  (SA).  As  noted  earlier,  the 
Letter  Factory  Test  contained  multiple-choice  ques¬ 
tions  designed  to  measure  three  levels  of  SA.  Fourteen  of 
these  items  were  designed  to  measure  SA  Level  1,  16 


items  to  measure  SA  Level  2,  and  1 4  items  to  measure  SA 
Level  3.  Table  3.3.17  summarizes  findings  from  the 
analyses  on  the  items  within  the  scale  for  each  of  these 
three  levels.  If  an  item  revealed  a  corrected  item-total 
correlation  (column  3)  of  less  than  .10,  it  was  removed 
from  the  scale.  This  reduced  the  number  of  Level  1, 
Level  2,  and  Level  3  items  to  8,  11,  and  7,  respectively. 
A  reliability  analysis  on  the  remaining  SA  items  showed 
alphas  on  the  three  new  scales  of  .42,  .61,  and  .47. 

Next,  a  scale  score  was  computed  for  each  of  the  three 
SA  levels.  These  scores  were  used  in  a  reliability  analysis 
to  determine  whether  the  three  scales  were  independent 
or  could  be  combined  into  one  scale  that  measured  the 
general  construct  of  SA.  The  alpha  computed  on  the 
three  scale  scores  was  .53.  The  results  indicated  that 
removal  of  any  one  of  the  three  scale  scores  would  not 
increase  the  alpha.  These  results  supported  the  notion 
that  all  remaining  SA  items  should  be  combined  into 
one  scale. 

Table  3.3.18  presents  findings  from  a  reliability 
analysis  on  the  26  remaining  SA  items  scored  as  one 
scale.  The  alpha  computed  on  this  overall  scale  was  .68. 
The  corrected  item-total  correlations  computed  in  the 
reliability  analysis  on  the  three  separate  SA  scales  (Table 
3.3. 17,  “After  Item  Deletion”)  were  very  similar  to  the 
corresponding  corrected  item-total  correlations  com¬ 
puted  for  the  combined  scale  (Table  3.3.18).  This  also 
supports  the  notion  of  using  a  single  number-correct 
score  across  all  remaining  SA  items. 

Analysis  of  LFT  Retest 

The  LFT  Form  B  or  retest  contains  five  test  sequences 
that  are  parallel  to  the  following  sequences  in  the  LFT 
Form  A:  3,  6,  7,  10,  and  11.  Form  B  contains  37  P/T 
items,  54  RI  items  (27  pre-interruption  items  and  27 
post-interruption  items),  and  20  SA  items. 

Case  Elimination 

First,  we  identified  participants  who  had  been  classed 
as  unmotivated  or  random  responders  while  taking 
Form  A  of  the  LFT.  Twenty-three  of  those  participants 
also  received  Form  B  of  the  LFT.  Since  Form  B  was 
designed  to  serve  as  a  retest,  we  eliminated  the  Form  B 
23  cases  that  had  been  eliminated  from  Form  A. 

Next,  the  same  criteria  used  in  Form  A  were  used  to 
eliminate  unmotivated  participants  and  inappropriate 
responders  in  Form  B.  To  look  for  unmotivated  partici¬ 
pants,  we  considered  performance  on  the  number  cor¬ 
rect  across  all  P/T  items  in  the  test  sequences.  Table 
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3.3.19  provides  an  overview  of  the  distribution  of 
those  number-correct  scores  across  the  entire  sample 
of  217  cases. 

A  natural  gap  was  evident  in  the  distribution  and 
cases  where  the  number-correct  score  was  lower  than  30 
were  eliminated.  Then,  participants  who  were  respond¬ 
ing  inappropriately  to  items  were  identified.  During  the 
test,  there  were  37  times  when  a  participant  should  have 
placed  a  box  in  the  loading  area.  Table  3.3.20  provides 
an  overview  of  the  distribution  of  the  number  of  inap¬ 
propriate  attempts  to  place  a  box  in  the  loading  area 
across  the  entire  sample  of  217  cases.  Cases  where  the 
inappropriate  response  score  was  higher  than  37  were 
eliminated.  Aftet  omitting  cases  based  on  the  above 
criteria,  the  sample  size  for  this  test  was  reduced  from 
217  to  184. 

Item  Elimination 

Again,  since  Form  B  was  designed  to  serve  as  a  retest, 
the  findings  from  analyses  performed  on  LFT  Form  A 
were  used  to  determine  which  test  items  to  eliminate 
from  Form  B.  We  removed  8  SA  items  so  12  SA  items 
remain  in  Form  B.  Similarly,  a  P/T  score  was  computed 
for  Form  B  by  subtracting  the  number  of  unnecessary 
mouse  clicks  from  the  number-correct  score  across  all  P/ 
T  items. 

Performance  Differences 

Form  B  was  used  to  assess  whether  participants  had 
reached  an  asymptote  of  performance  during  Form  A. 
Different  sequences  could  not  be  used  in  Form  A  for  this 
test  because  the  item  types  are  very  heterogeneous,  and 
little  information  is  available  on  item  or  sequence 
difficulties.  By  matching  test  sequences,  we  can  control 
for  manageable  aspects  of  the  test  that  impact  test 
performance.  Table  3.3.21  presents  the  results  of  depen¬ 
dent  r-tests  comparing  two  performance  measures.  Those 
results  show  no  support  for  a  change  in  participants’ 
performance  on  Situational  Awareness.  However,  the 
roughly  8%  performance  increment  on  Planning  and 
Thinking  Ahead  was  a  significant  increase  in  perfor¬ 
mance.  This  suggests  that  participants  would  benefit 
from  more  practice  before  beginning  the  test. 

Time  Limit  Analyses 

Table  3.3.22  presents  tbe  distribution  of  test  comple¬ 
tion  times  for  the  LFT,  showing  that  95%  of  partici¬ 
pants  completed  the  LFT  in  64.9  minutes  or  less.  When 
we  use  the  normal  curve  to  compute  the  95th  percentile 
(i.e.,  take  1.96  times  the  standard  deviation  (7.63)  and 


add  that  product  to  the  mean),  we  estimate  a  slightly 
higher  amount  of  time  (66.7  minutes)  for  the  95th 
percentile  participant.  A  test  completion  time  of  67 
minutes,  then,  seems  appropriate  for  the  test  at  its 
current  length.  Of  this,  95%  of  participants  completed 
the  LFT  test  sequences  in  27.1  minutes.  This  leaves 
about  39.9  minutes  for  instructions  and  practice. 

The  Spearman-Brown  formula  was  used  to  estimate 
the  number  of  items  needed  to  raise  the  reliability  of 
Situational  Awareness  to  .80  (49  items)  and  .90  (110 
items).  Because  the  measure  of  Planning  and  Thinking 
Ahead  already  has  a  higher  estimated  reliability  of  .86, 
it  would  automatically  go  up  to  a  sufficient  level  when 
the  numbet  of  sequences  is  raised  to  increase  the  reliabil¬ 
ity  of  Situational  Awareness.  Table  3.3.23  presents  a 
recommended  composition  of  sequence  lengths  and 
number  of  Situational  Awareness  test  items  per  se¬ 
quence.  It  was  estimated  that  this  recommended  com¬ 
position  would  yield  a  test  reliability  in  the  low  .90s  for 
Planning  and  Thinking  Ahead,  and  a  reliability  in  the 
low  .80s  for  Situational  Awareness.  Based  on  experience 
with  the  alpha  LFT,  it  was  estimated  that  participants 
would  spend  45  minutes  on  the  test  portion  of  the  new 
test  version. 

The  amount  of  practice  before  the  test  also  needed  to 
be  increased.  The  initial  practice  sequence  was  an  easy 
30-second  sequence,  followed  by  sequences  of  2  min¬ 
utes  and  2.25  minutes.  Adding  three  more  sequences  of 
the  same  difficulty  as  the  test,  together  with  four  SA 
questions  after  each  sequence,  was  proposed.  One  30- 
second  sequence  and  two  2.5-minute  sequences  would 
add  an  additional  7.5  minutes  to  the  practice  and 
instruction  time.  Improving  the  instructions  to  empha¬ 
size  the  penalty  that  occurs  for  error  clicks  on  the 
Planning/Thinking  Ahead  measure  would  add  about 
half  a  minute.  Instruction  and  practice  time,  then, 
should  increase  by  8  minutes  from  39.9  to  about  48 
minutes.  With  a  45-minute  time  for  the  test  sequences, 
this  amounted  to  93  minutes  for  the  recommended  beta 
version  of  the  LFT. 

Test  Revisions 

Test  Sequences.  To  reduce  the  number  of  inappro¬ 
priate  responses  by  participants  who  double-click  or 
continuously  click  on  the  box  stack,  the  associated  error 
signal  (one  red  arrow)  was  changed  to  an  error  message. 
The  error  message  appears  in  red  above  the  box  stack 
when  participants  try  to  move  a  box  to  the  loading  area 
when  one  is  not  needed.  The  new  error  message  reads, 
“You  did  not  need  to  move  a  box.” 
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T o  increase  visibility,  the  error  signal  was  also  changed 
to  an  error  message  when  participants  try  to  place  a  letter 
in  a  box  that  is  not  the  fullest.  This  error  message  reads, 
“You  did  not  place  the  letter  in  the  fullest  box.” 

Analyses  described  above  showed  that  the  mean 
scores  for  RI  items  had  a  ceiling  effect.  Also,  it  was 
indicated  earlier  that  the  difficulty  level  of  the  RI  items 
could  not  be  increased  without  making  psychomotor 
ability  a  significant  component  of  the  task.  For  these 
reasons,  all  RI  items  were  removed  from  the  test. 

To  increase  test  reliability  for  the  P/T  and  SA  mea¬ 
sures,  the  number  of  test  sequences  and  SA  questions 
was  increased.  Test  sequences  were  increased  from  1 1  to 
1 8.  Level  1  SA  questions  were  increased  from  14  to  26, 
Level  2  SA  questions  from  16  to  24,  and  Level  3  SA 
questions  were  increased  from  14  to  26.  SA  questions 
also  were  revised. 

T est  Instructions.  Based  on  feedback  from  the  project 
team  and  the  FAA,  several  sections  of  the  instructions 
were  changed.  The  pilot  version  of  the  test  included 
several  short  practice  sequences  in  the  middle  of  the 
instructions  and  three  complete  practice  sequences  at 
the  end  of  the  instructions  but  before  the  test.  To 
increase  the  amount  of  practice,  two  practice  sequences 
were  added  in  the  middle  of  the  instructions,  allowing 
participants  to  practice  moving  boxes  to  the  loading  area 
and  placing  letters  into  boxes.  In  addition,  the  mouse 
exercise  appears  before  the  practice  sequences  so  partici¬ 
pants  can  learn  how  to  use  the  mouse  before  they 
practice  taking  the  test.  In  addition,  the  mouse  exercise 
was  changed  so  participants  can  choose  to  receive  addi¬ 
tional  practice  using  the  mouse  (up  to  nine  trials),  or 
move  to  the  next  set  of  instructions. 

Other  changes  in  response  to  project  team  and  FAA 
feedback  include  (a)  not  showing  the  mouse  cursor 
(arrow)  on  the  screens  in  which  the  mouse  is  disabled; 
(b)  adding  a  screen  that  identifies  the  six  items/ areas  that 
make  up  the  test  display;  (c)  simplifying  and  increasing 
the  speed  of  some  of  the  examples;  (d)  changing  the 
“Call  Quality  Control”  screen  label  to  “Quality  Con¬ 
trol”;  (e)  and  simplifying  some  words  and  sentences  in 
portions  of  the  instructional  text. 

Changes  were  also  made  in  parts  of  the  instructions 
in  response  to  the  data  analyses.  To  reduce  the  number 
of  inappropriate  responses  due  to  double-clicking  or 
constant  attempts  to  place  boxes  in  the  loading  area, 
instructions  were  added  telling  participants  to  click  on 
the  mouse  only  once  and  to  not  double-click.  Corrective 
feedback  also  was  added  to  the  practice  sequences  that 
appear  in  the  middle  of  the  instructions.  For  these 


practice  sequences,  if  a  participant  clicks  more  than 
once  to  move  a  box  to  the  loading  area,  the  screen  freezes 
and  the  following  corrective  feedback  appears:  “To 
move  a  box  to  the  loading  area,  you  only  need  to  click 
on  the  box  once.  Do  not  double-click.”  The  following 
corrective  feedback  appears  when  a  participant  does  not 
place  a  box  in  the  loading  area  when  one  is  needed:  “The 
computer  moved  a  box  into  the  loading  area  for  you. 
Keep  track  of  new  letters  that  appear  on  the  conveyor 
belt  and  move  boxes  to  the  loading  area  as  soon  as  they 
are  needed.”  If  the  computer  places  a  letter  into  a  box  for 
the  participant,  the  following  corrective  feedback  ap¬ 
pears:  “A  letter  fell  off  a  conveyor  belt  and  the  computer 
placed  it  in  a  box.  Make  sure  you  track  the  letters  as  they 
move  down  the  belts.”  The  last  corrective  feedback 
appears  when  a  participant  tries  to  place  a  letter  in  a  box 
that  is  not  closest  to  being  full:  “You  need  to  place  the 
letters  in  the  boxes  that  are  closest  to  being  full.” 

Finally,  since  the  RI  measure  had  been  eliminated 
from  tbe  test,  any  reference  to  RI  was  removed  from  the 
instructions.  Also,  in  view  of  the  significant  increase  in 
participant  performance  on  the  P/T  measure  on  the 
retest,  three  practice  sequences  were  added  at  the  end  of 
the  instructions.  This  provides  participants  with  a  total 
of  six  practice  sequences  at  the  end  of  the  instructions. 

Situational  Awareness  Items.  SA  items  at  the  end  of 
each  LFT  sequence  were  revised  on  the  basis  of  the  item 
analyses  described  above.  The  following  subsections 
below  provide  an  overview  of  how  item  analyses  guided 
development  of  new  SA  items. 

For  all  levels  of  SA,  care  was  taken  to  not  ask  any 
questions  about  letters  below  availability  lines  on  the 
belts.  The  reason  was  that  examinees  who  worked 
quickly  would  have  completed  tasks  associated  with 
letters  below  availability  lines;  for  such  examinees,  those 
letters  might  not  even  be  on  the  belts,  but  rather  in 
boxes.  For  all  examinees,  though,  the  letters  above  the 
belts  would  all  be  in  the  same  places. 

Level  1  Items.  Level  1  Situational  Awareness  items 
assess  an  examinee’s  ability  to  perceive  elements  in  the 
environment,  together  with  their  status,  attributes,  and 
dynamics.  The  following  are  examples  of  item  stems 
(i.e.,  the  questions)  that  item  analysis  supported: 

■  Which  belt  moved  letters  the  SLOWEST? 

•  Which  belt  had  the  most  letters  ABOVE  the  availabil¬ 
ity  line? 

•  Which  two  belts  have  their  availability  lines  closest  to 
the  BOTTOM  of  the  belt? 
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•  How  many  ORANGE  letters  were  ABOVE  the  avail¬ 
ability  lines? 

•  Which  belt  had  no  letters  on  it? 

Listed  below  are  examples  of  item  stems  that  item 
analysis  did  NOT  support: 

■  Which  belt  moved  letters  the  EASTEST? 

•  Which  belt  had  its  availability  line  CLOSEST  to  the 
TOP  of  the  belt? 

•  Which  letter  was  CLOSEST  to  the  TOP  of  the  belt? 

■  Which  letter  was  CLOSEST  to  crossing  the  availabil¬ 
ity  line? 

Item  analysis,  therefore,  suggested  that  Level  1  SA  is 
more  reliably  measured  in  the  LET  by  asking  about  the 
number  and  color  of  letters  above  the  availability  lines, 
which  belt  was  slowest,  and  which  belts  had  their 
availability  lines  closest  to  the  bottom.  These  questions 
are  consistent  with  targeting  items  toward  a  lower 
average  level  ofplanning  and  thinking  ahead.  Generally, 
the  items  that  did  not  contribute  well  to  scale  reliability 
included  those  about  the  fastest  belt,  availability  lines 
closer  to  the  top  of  the  belt,  and  letters  closer  to  the  top 
of  the  belt.  Those  items  are  more  consistent  with  a 
higher  level  of  planning  and  thinking  ahead.  Therefore, 
development  of  new  Level  1  SA  items  was  focused  on  the 
types  of  areas  listed  above  that  required  lower  amounts 
of  planning  and  thinking  ahead. 

Level  2  Items.  Level  2  Situational  Awareness  items 
assess  an  examinee’s  ability  to  comprehend  the  situa¬ 
tion.  This  requires  a  synthesis  of  disjointed  Level  1 
elements.  In  asking  about  Level  2  SA,  it  is  important  to 
assess  comprehension  of  Level  1  elements  while  requir¬ 
ing  no  projection  of  the  future  status  (a  Level  3  factor). 
In  the  LET  setting,  it  was  also  considered  necessary  to 
clearly  distinguish  between  letters  already  in  boxes  in  tbe 
loading  areas,  compared  with  letters  that  examinees 
could  place  there.  To  make  this  distinction  clear,  short 
(i.e.,  30-second)  scenarios  where  no  letters  ever  crossed 
any  availability  lines  were  developed.  That  way,  no 
letters  were  in  boxes  at  the  end  of  these  short  scenarios. 
In  these  scenarios,  examinees  could  only  place  boxes  and 
maintain  their  situational  awareness. 

The  following  are  examples  of  Level  2  item  stems 
(i.e.,  the  questions)  that  item  analysis  supported: 

•  What  color  was  the  LAST  box  you  should  have  placed 
in  the  loading  area  in  order  to  correctly  place  all  the 
letters  into  boxes? 


•  Consider  tbe  LAST  box  you  should  have  placed  in  the 
loading  area.  Which  letter  caused  you  to  need  to  place 
this  last  box? 

•  How  many  (or  which)  boxes  should  be  in  the  loading 
area  in  order  to  correctly  place  all  the  letters? 

•  Consider  all  the  letters  on  the  belts.  What  color  were 
the  letters  that,  when  combined,  could  fill  at  least  one 
box? 

Listed  below  are  examples  of  item  stems  that  item 
analysis  did  NOT  support: 

•  If  all  the  letters  were  correctly  placed  in  boxes,  how 
many  empty  spaces  for  Ds  would  be  in  the  boxes  in  the 
loading  area? 

•  If  all  letters  were  correctly  placed  in  boxes,  how  many 
more  (or  which)  letters  would  be  needed  to  completely 
fill  the  GREEN  box? 

Item  analysis,  therefore,  suggested  that  Level  2  SA  is 
more  reliably  measured  in  the  LET  by  asking  about  the 
last  box  an  examinee  should  have  placed  in  the  loading 
area,  the  number  or  color  of  boxes  that  should  be  in  the 
loading  area,  and  what  color  of  letters  could  completely 
fill  a  box.  These  questions  are  consistent  with  targeting 
items  toward  the  more  immediate  LFT  concerns.  Gen¬ 
erally,  the  items  that  did  not  contribute  well  to  Level  2 
scale  reliability  included  those  about  the  number  of 
empty  spaces  for  a  particular  letter,  and  how  many  more 
or  which  letters  were  required  to  fill  a  particular  color  of 
box.  Those  items  are  more  consistent  with  a  more  fine¬ 
grained  level  of  situational  awareness.  Therefore,  devel¬ 
opment  of  new  Level  2  SA  items  was  focused  on  the 
types  of  areas  listed  above  that  required  only  an  aware¬ 
ness  of  the  more  immediate  LFT  concerns. 

Level  3  Items.  Level  3  SA  items  assess  an  examinee’s 
ability  to  project  tbe  future  status  or  actions  of  the 
elements  in  the  environment.  This  requires  a  knowledge 
of  the  status  and  dynamics  of  the  elements,  as  well  as  a 
comprehension  of  the  situation — both  Level  1  and 
Level  2  SA.  In  asking  about  Level  3  SA  for  LFT 
sequences,  it  is  important  to  ensure  that  all  examinees 
are  responding  to  the  same  situation.  Although  the 
software  stops  at  exactly  the  same  place  for  all  examin¬ 
ees,  the  quicker  examinees  may  have  placed  more  boxes 
in  the  loading  area  or  placed  more  letters  in  boxes.  For 
this  reason,  all  Level  3  SA  items  were  started  with  the 
following  two  sentences  separated  as  a  paragraph  before 
the  Level  3  question: 
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“Assume  that  you  correctly  placed  all  required  boxes  in  the 
loading  area.  Also  assume  that  you  correctly  placed  all  the 
letters  that  remained  on  the  belts.” 

The  intent  of  this  introduction  was  to  allow  all 
examinees  to  (at  least  mentally)  begin  from  the  same 
situation.  Item  analysis  showed  that  Level  3  SA  is  more 
reliably  measured  in  the  LFT  by  asking  about  simple 
situations.  In  the  case  of  the  LFT,  that  seemed  to  be  a 
situation  having  only  two  or  three  boxes  in  the  loading 
area.  The  following  are  examples  of  Level  3  item  stems 
(i.e.,  the  questions)  that  item  analysis  supported  for 
those  simple  situations: 

•  After  the  full  boxes  are  removed,  which  boxes  would 
remain  in  the  loading  area? 

•  Which  letters  would  you  need  to  complete  an  OR¬ 
ANGE  box? 

•  How  many  more  letters  would  you  need  to  fill  all  the 
boxes  remaining  in  the  loading  area? 

•  If  the  next  letter  was  a  PURPLE  A,  which  of  the 
following  would  be  true? 

Therefore,  development  of  new  Level  3  SA  items  was 
focused  on  simple  situations  and  used  the  types  of 
questions  listed  above. 

Summary  and  Recommendations 
The  original  plan  was  to  measure  three  worker  re¬ 
quirements  using  the  LFT.  Because  the  measure  of 
Recall  From  Interruption  showed  ceiling  effects  and 
unreliable  difference  scores,  it  was  recommended  that 
attempts  to  measure  that  worker  requirement  with  this 
test  be  abandoned.  To  more  adequately  measure  the 
worker  requirements  of  Planning  and  Thinking  Ahead 
and  Situational  Awareness,  lengthening  the  test  to  93 
minutes  was  recommended.  This  longer  version  in¬ 
cludes  doubling  the  number  of  practice  sequences  that 
participants  complete  before  they  begin  the  test.  It  was 
estimated  that  this  extra  practice  would  reduce  the 
practice  effect  observed  between  the  LFT  and  the  retest 
LFT  on  a  small  (N  =  184)  subsample.  This  would  help 
ensure  that  participants  perform  at  or  near  their  ability 
prior  to  beginning  the  test  portion  of  the  LFT. 

Scan  Test 

Data  Collection/Soffware  Problems 
As  the  data  collection  proceeded  on  the  Scan  test,  it 
became  clear  that  the  software  was  not  writing  data  for 
change  items  nor  was  it  recording  item  latencies.  A 


correction  to  the  software  was  implemented  on  Febru¬ 
ary  26.  Of  the  429  cases  on  which  data  were  collected, 
151  cases  had  complete  data. 

Case  Elimination 

Because  all  participants  proceed  at  the  same  rate 
during  practice  and  test  sequences,  test  completion  time 
could  not  be  used  to  assess  participants’  test-taking 
motivation.  Likewise,  because  the  test  software  auto¬ 
matically  writes  out  data  for  each  item  indicating  whether 
the  participant  correctly  selected  the  item,  no  cases 
should  have  missing  data. 

Unmotivated  Participants.  It  was  believed  that  un¬ 
motivated  participants  would  respond  to  very  few  or 
none  of  the  items,  or  respond  with  irrelevant  answers. 
The  number-correct  scores  were  used  to  identify  unmo¬ 
tivated  participants.  The  distribution  of  the  429  partici¬ 
pants  is  provided  in  Table  3.3.24.  An  examination  of 
tbe  data  showed  that  no  participant  simply  sat  at  the 
computer  and  allowed  the  software  to  progress  on  its 
own.  Each  participant  entered  some  appropriate  re¬ 
sponses,  and  each  got  at  least  a  few  items  correct.  The 
lowest  score  shown  was  22  out  of  the  162  questions 
correct  (13.6%).  While  there  may  have  been  partici¬ 
pants  who  were  not  trying  their  best,  this  screening 
algorithm  was  unable  to  identify  participants  who  bla¬ 
tantly  did  nothing  at  all.  Therefore,  all  cases  were  kept 
for  the  analyses. 

Item  Analyses 

Table  3.3.25  presents  findings  from  the  reliability 
analysis  on  the  four  test  sequences  (i.e.,  Tl  to  T4).  The 
three  parts  of  the  table  show  how  the  sequence  reliabilities 
measured  by  alpha  differed  as  different  groups  of  items 
were  deleted.  The  first  part  (“With  Change  Items”) 
presents  results  that  include  all  the  items  in  each  se¬ 
quence.  Each  change  item  may  be  considered  as  two 
items;  the  item  is  what  was  presented  originally,  and  the 
second  is  the  item  with  the  change  in  the  bottom  or 
three-digit  number.  The  middle  columns  include  the 
pre-change  items  and  exclude  the  post-change  items, 
and  the  third  part  of  the  table  removes  both  versions  of 
the  change  items  (i.e.,  the  original  and  the  change  part). 
Notice,  too,  that  the  second  and  third  parts  of  the  table 
show  “Actual”  and  “Expected”  alphas.  The  actual  al¬ 
phas  are  the  results  provided  by  the  data.  The  expected 
alphas  are  the  ones  estimated  by  the  Spearman-Brown 
formula  if  “like”  items  were  deleted.  In  every  case,  the 
alphas  from  the  data  are  higher  than  the  expected  alphas. 
This  finding  supports  the  notion  that  the  change  items 
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differ  from  the  other  items  in  the  test.  Not  including 
them  in  the  scoring,  therefore,  should  increase  the  alpha 
test  reliability. 

Of  the  166  remaining  items  in  sequences  Tl  to  T4, 
only  four  items  (i.e.,  items  359, 373, 376,  and  4l  0)  had 
item-total  correlations  less  than  .10.  The  alpha  com¬ 
puted  on  the  162  items  remaining  was  .959.  This 
supported  computing  a  number  correct  score  using 
these  162  items  for  the  scanning  worker  requirement. 

Time  Limit  Analyses 

Table  3.3.26  shows  the  distribution  of  test  times  for 
participants  in  this  sample;  95%  completed  the  Scan 
Test  in  19.87  minutes  or  less.  If  we  take  1.96  times  the 
standard  deviation  of  test  completion  times  (1.75)  and 
add  that  product  to  the  mean  test  completion  time 
( 1 6.92),  we  find  that  a  95th  percentile  participant  might 
take  20.35  minutes  to  complete  the  Scan  Test.  Due  to 
the  obtained  test  reliability,  it  was  recommended  that 
no  change  be  made  to  the  test  time  for  the  Scan  test,  with 
21  minutes  allocated  for  this  test. 

Summary  and  Recommendations 

Items  in  the  Scan  test  that  change  during  their  screen 
presentation  did  not  behave  the  same  as  othet  items  in 
the  test.  Eliminating  those  items  improved  estimates  of 
internal  consistency  reliability.  After  eliminating  four 
items  that  had  poor  item-total  correlations,  the  162 
remaining  items  in  the  test  (i.e.,  non-practice)  portion 
of  the  Scan  test  produced  an  alpha  of  .96.  Therefore,  we 
recommend  keeping  the  Scan  test  at  its  curtent  length 
and  allocating  21  minutes  for  test  completion. 

Participant  performance  on  the  Scan  test  items  is 
affected  by  the  configuration  of  other  items  ptesented 
on  the  same  screen  so  any  change  must  be  considered 
carefully.  As  a  class  of  items,  the  change  items  tended  to 
reduce  the  Scan  test  reliability.  By  eliminating  the 
changing  nature  of  the  change  items,  the  test  instruc¬ 
tions  could  be  simplified.  However,  eliminating  those 
items  might  make  the  test  easier  or  change  the  test  in 
some  other  way.  Therefore,  it  was  recommended  to  keep 
the  items  as  they  are  presented  initially  (i.e.,  without  the 
changing  feature)  but  not  count  them.  A  similar  recom¬ 
mendation  was  made  for  the  four  items  that  showed 
poor  item-total  correlations. 


Planes  Test 

Case  Elimination 

The  Planes  test  consisted  of  three  parts  and  cases  were 
eliminated  from  each  part  independently.  The  screen¬ 
ing  algorithms  for  each  part  were  based  on  similar 
premises. 

Part  1  consisted  of  48  items.  Participants  were  elimi¬ 
nated  from  further  analyses  if  any  of  three  screening 
criteria  were  satisfied.  The  first  screen  for  this  part  was 
a  total  latency  less  than  48  seconds.  The  second  screen 
was  percent  correct  less  than  or  equal  to  40%.  The  final 
screen  was  the  skipping  of  six  or  more  items.  These 
screening  algorithms  reduced  the  sample  from  450  to 
429  for  Part  1. 

The  screening  for  Part  2  was  similat.  Participants 
were  eliminated  from  further  analyses  on  these  criteria: 
(1).  Part  2  total  latency  less  than  1.2  minutes,  (2).  40% 
correct  or  less,  or  (3).  missing  data  for  six  or  more  items. 
These  screening  algorithms  reduced  the  available  sample 
from  450  to  398  for  Part  2. 

Part  3,  participants  were  eliminated  on  these  criteria: 
(1)  Part  3  total  latency  less  than  2.4  minutes,  (2)  40% 
correct  or  less,  or  (3).  missing  data  for  1 2  or  more  items. 
These  screening  algorithms  reduced  the  available  sample 
from  450  to  366  for  Part  3. 

Participant  elimination  across  all  three  test  parts  left 
a  final  sample  of  343  having  data  on  all  three  parts. 

Item  Analyses 

Scale  Reliabilities  and  Item  Analyses.  Reliability 
analyses  were  conducted  to  identify  items  within  each 
part  of  the  Planes  test  that  contribute  to  internal  consis¬ 
tency.  The  corrected  item-total  correlation  was  com¬ 
puted  for  each  item  within  each  part  as  was  the  overall 
alpha  for  that  part.  Table  3.3.27  presents  an  overview  of 
the  results  of  these  reliability  analyses. 

The  Planes  test  is  not  a  new  test,  having  been  devel¬ 
oped  previously  as  the  Ships  test  (Schemmer  et  ah, 
1996).  In  its  alpha  text  form,  the  number  of  items  was 
cut  in  half  to  meet  the  time  allowed  for  it  in  the  pretest. 
In  reducing  the  number  of  items,  the  same  proportion 
was  kept  for  all  item  types.  However,  there  are  many 
parallels  between  the  items  in  each  of  the  three  parts  of 
the  test;  a  particular  item  that  may  not  work  well  in  Part 
1  mightworkveiywellinParts2or3.  For  these  reasons  and 
because  data  from  all  three  parts  were  to  be  used  to  develop 
a  residual  score  for  the  coordinating  ability  component  of 
multitasking,  eliminating  items  based  on  poor  item-total 
correlations  alone  was  not  considered  desirable. 
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Restoring  the  Planes  test  to  its  original  length  would 
require  doubling  the  number  of  items.  Using  the  Spearman- 
Brown  formula,  the  new  reliabilities  are  estimated  at  .86  for 
Part  1,  .91  for  Part  2,  and  .89  for  Part  3. 

Computing  Residual  Scores.  Using  number  correct 
scores  from  Planes  Part  1  and  Part  2,  the  regression 
procedure  outlined  in  Yee,  Hunt,  and  Pellegrino  (1991) 
was  followed  to  compute  an  estimate  of  the  coordinat¬ 
ing  ability  component  of  multitasking.  First,  the  regres¬ 
sion  equation  for  predicting  the  Part  3  score  was 
computed.  Then,  the  difference  between  the  actual  and 
predicted  scores  for  Part  3  was  computed  by  subtracting 
the  predicted  from  the  actual  score.  This  residual  score 
estimates  the  coordinating  ability  aspect  of  multitasking. 

Yee  et  al.  argue  that  a  necessary  but  not  sufficient 
condition  for  the  residual  scores  to  be  useful  is  that  they 
must  be  reliable.  As  they  indicate,  the  quantity  (1-R^) 
must  be  greater  than  zero  allowing  for  unreliability 
in  the  performance  measures  involved.  To  show  the 
residual  score  as  reliable,  analysts  corrected  the  test 
scores  for  each  of  the  three  parts  of  the  Planes  test  for 
unreliability  and  created  a  new  correlation  matrix.  Us¬ 
ing  this  corrected  correlation  matrix,  the  multiple  cor¬ 
relation  was  computed  to  predict  Part  3  from  Parts  1  and 
2  (R  =  .506,  R^  =  .256).  To  the  extent  that  this  multiple 
R^  is  less  than  unity  after  correcting  all  performance 
measures  for  unreliability,  the  residual  scores  may  be 
considered  reliable. 

In  addition,  analysts  followed  the  procedure  of  Yee  et 
al.  and  compared  the  multiple  R^  (computed  on  ob¬ 
served  scores,  R^  = .  1 64)  to  the  reliability  of  Part  3  (alpha 
=  .804).  Both  analyses  supported  the  inference  of  re¬ 
sidual  score  reliability.  Finally,  we  used  the  reliabilities 
of  the  predicted  and  actual  scores  to  estimate  the  reli¬ 
ability  of  the  residual  score  (r  =  .6 1 3) .  The  reliability  of 
the  coordinating  ability  score  for  a  Planes  test  of  twice 
the  length  was  estimated  to  be  .65. 

Time  Limit  Analyses 

Table  3.3.28  shows  the  distribution  of  test  comple¬ 
tion  times  for  the  Planes  test.  Ninety-five  per  cent  of 
participants  completed  the  Planes  test  in  34.6  minutes 
or  less.  When  we  take  1 .96  times  the  standard  deviation 
(4.42)  and  add  that  product  to  the  mean,  we  estimate  a 
slightly  higher  amount  of  time  (36.4  minutes).  A  test 
completion  time  of  37  minutes,  then,  seems  appropriate 
for  the  test  at  its  current  length.  Of  this  37  minutes,  95% 
of  participants  completed  instructions  and  practice  in 
21.6  minutes.  This  leaves  15.4  minutes  for  completing 


the  192  items  in  the  three  test  parts.  Doubling  the 
number  of  items,  then,  would  increase  the  test  time  by 
15.4  minutes,  from  37  to  52  minutes. 

Test  Revisions 

Following  the  alpha  testing  of  the  Air  Traffic  Con¬ 
troller  Test,  the  Planes  test  was  revised  in  several  ways, 
including  test  and  practice  length,  test  instructions, 
response  mode,  and  content. 

Test  Length.  Part  3  was  reduced  to  48  from  96 
questions,  the  one-minute  breaks  were  cut  to  30  sec¬ 
onds,  and  practice  sessions  were  reduced  from  24  to  12 
questions. 

Mode  of  Response.  The  mode  of  response  was 
changed  for  all  three  subtests.  Parts  1  and  2  were 
changed  to  keys  labeled  R  for  the  red  plane  and  W  for 
the  white  plane  instead  of  the  numeric  keypad  “  1  ”  key 
to  represent  the  red  plane  and  “3”  key  on  the  numeric 
keypad  to  represent  the  white  plane.  Part  3  changed  to 
keys  labeled  T  for  true  and  F  for  false,  instead  of  using 
“  1  ”  and  “3”  of  the  numeric  keypad  to  represent  false  and 
true,  respectively. 

Test  Content  The  content  of  Part  3  was  changed  so 
that  all  questions  used  “double-negative”  statements 
(e.g.,  “It  is  not  true  that  the  white  plane  will  not  arrive 
after  the  red  plane.”),  thereby  making  Part  3  distinct 
from  Part  2.  Previously,  some  questions  in  Part  3  were 
like  those  in  Part  2. 

Instructions.  The  test  instructions  were  simplified  in 
several  places.  Also,  the  instructions  in  the  “Answering 
Test  Items”  section  were  revised  to  correspond  to  the 
changes  made  in  mode  of  response  (noted  above). 

Summary  and  Recommendations 

The  project  team  cut  the  number  of  items  in  each  part 
of  the  original  Planes  test  in  half  for  the  alpha  data 
collection  effort.  This  was  done  to  meet  project  time 
constraints.  After  completing  reliability  analyses,  it  was 
clear  that  the  test  would  benefit  from  restoring  it  to  its 
original  length.  Available  test  time  in  the  beta  version 
was  limited,  however.  As  a  result,  the  number  of  items 
in  Part  3  and  in  the  practice  sessions  was  cut  in  half.  The 
time  allotted  for  breaks  between  the  three  test  parts  was 
also  halved. 

Experiences  Questionnaire 

The  following  Experiences  Questionnaire  analyses 
were  performed  on  data  from  the  first  9  of  the  12  days 
of  pilot  testing  at  Pensacola  in  February,  1 997.  The  total 
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N  in  this  data  set  is  330.  The  last  2  days  of  pilot  testing 
included  a  large  number  of  the  ATCS  students;  perfor¬ 
mance  of  the  ATCS  students  and  performance  of  Non- 
ATCS  students  on  the  EQhave  not  been  compared. 

EQ  Format 

The  pilot  test  version  of  the  EQ  contained  201  items 
representing  17  scales,  including  a  Random  Response 
Scale.  All  items  used  the  same  set  of  five  response 
options:  Definitely  True,  Somewhat  True,  Neither 
True  Nor  False,  Somewhat  False,  and  Definitely  False. 

Data  Screening 

Three  primary  data  quality  screens  are  typically  per¬ 
formed  on  questionnaires  like  the  EQ:  (a)  a  missing  data 
screen,  (b)  an  unlikely  virtues  screen,  and  (c)  a  random 
response  screen.  The  missing  data  rule  used  was  that  if 
more  than  10%  of  the  items  on  a  particular  scale  were 
missing  (blank),  that  scale  score  was  not  computed.  No 
missing  data  rule  was  invoked  for  across-scale  missing 
data,  so  there  could  be  a  data  file  with,  for  example,  all 
scale  scores  missing.  No  one  was  excluded  based  on 
responses  to  the  unlikely  virtues  items,  that  is,  those 
items  with  only  one  "likely”  response  (Example:  “You 
have  never  hurt  someone  else’s  feelings,”  where  the  only 
“likely”  response  is  “Definitely  False”). 

A  new  type  of  random  response  item  was  tried  out  in 
the  pilot  test,  replacing  the  more  traditional,  right/ 
wrong-answer  type,  such  as  “Running  requires  more 
energy  than  sitting  still.”  There  were  four  random 
response  items,  using  the  following  format:  “This  item 
is  a  computer  check  to  verify  keyboard  entries.  Please 
select  the  Somewhat  T rue  response  and  go  on  to  the  next 
item.”  The  response  that  individuals  were  instructed  to 
select  varied  across  the  four  items.  A  frequency  distribu¬ 
tion  of  the  number  of  random  responses  (responses 
other  than  the  correct  one)  follows: 


Number  of 

Random 

Responses 

N 

Percent 

0 

222 

67.3 

1 

52 

15.8 

2 

34 

10.3 

3 

18 

5.5 

4 

A 

1.2 

330 

100.0 

As  can  be  seen,  a  large  number  of  students  gave  one 
or  more  random  responses  (108,  or  32.8%).  Whether 
this  indicates  that  the  new  random  response  items  are 
too  difficult,  or  that  a  large  number  of  students  were  not 
attending  very  closely  to  the  EQ  (or  other  tests?)  is 
unclear.  Students  with  two  or  more  random  responses 
were  removed  from  the  data  set,  resulting  in  a  screened 
sample  of  274  EQs  available  for  further  analysis. 

Time  to  Complete  EQ 

The  mean  amount  of  time  required  to  complete  the 
EQ  for  the  screened  data  set  was  29.75  minutes  (SD  = 
9.53,  Range  =  10-109).  A  few  individuals  finished  in 
approximately  1 0  minutes,  which  translates  into  roughly 
3  seconds  per  response.  The  records  of  the  fastest 
finishers  were  checked  for  unusual  response  patterns 
such  as  repeating  response  patterns  or  patterns  of  all  the 
same  response  (which  would  yield  a  high  random  re¬ 
sponse  score  anyway),  and  none  were  found.  Thus,  no 
one  was  deleted  from  the  data  set  due  solely  to  time 
taken  to  complete  the  test.  It  is  not  surprising  to  note 
that  the  fastest  finishers  in  the  entire,  unscreened  sample 
of 330  were  deleted  based  on  their  scores  on  the  random 
response  scale. 

Scale  Scoring 

EQ  items  were  keyed  1  -5,  the  appropriate  items  were 
reversed  (5  -  1),  and  the  scale  scores  were  computed  as 
(the  mean  item  response)  x  20,  yielding  scores  ranging 
from  20  to  1 00.  The  higher  the  score,  the  higher  the 
standing  on  the  characteristic. 

Descriptive  Statistics  and  Reliability  Estimates 

Appendix  B  contains  the  descriptive  statistics  and 
internal  consistency  reliabilities  for  16  scales  (Random 
Response  Scale  excluded).  The  scale  means  were  com¬ 
fortingly  low  and  the  standard  deviations  were  comfort¬ 
ingly  high,  relieving  concerns  about  too  little  variance 
and/or  a  ceiling  effect.  The  Unlikely  Virtues  scale  had 
the  lowest  mean  of  all  (51.85),  as  it  should. 

The  scale  reliabilities  were  within  an  acceptable  range 
for  scales  of  this  length  and  type.  Most  were  in  the  .70s 
and  .80s.  The  two  exceptions  were  Self  Awareness  (.55) 
and  Self-Monitoring/Evaluating  (.54). 

Four  items  had  very  low  item-scale  correlations,  so 
they  were  removed  from  their  respective  scales:  Items  2 1 
and  53  from  the  Decisiveness  scale  (item-scale  correla¬ 
tions  of -.02  and  -.05  respectively),  item  144  from  the 
Self-Monitoring/Evaluating  scale  (correlation  of  .04), 
and  item  163  from  the  Interpersonal  Tolerance  scale 
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(correlation  of  -.19).  Each  of  these  four  items  was 
correlated  with  all  of  the  other  scales  to  see  if  they  might 
be  better  suited  to  another  scale.  Item  144  correlates  .23 
with  the  Interpersonal  Tolerance  scale,  and  its  content 
is  consistent  with  that  scale,  so  it  has  been  moved.  The 
remaining  three  items  either  did  not  correlate  high 
enough  with  other  scales,  or  the  item  content  was  not 
suflficiently  related  to  the  other  scales  to  warrant  moving 
them.  These  three  items  were  deleted,  and  the  descrip¬ 
tive  statistics  and  internal  consistency  reliabilities  were 
rerun  for  the  three  scales  affected  by  the  item  deletions/ 
moves.  Appendix  B  contains  the  revised  descriptive 
statistics  and  internal  consistency  reliabilities  for  the 
three  scales  affected. 

At  the  item  level,  the  means  and  standard  deviations 
were  satisfactory.  (Item  means  and  SDs  can  be  found  in 
the  reliability  output  in  Appendix  B,  The  only  items 
with  extreme  values  and/or  low  standard  deviations  were 
on  the  Unlikely  Virtues  scale,  which  is  as  it  should  be. 

Scale  Intercorrelations  and  Factor  Analysis 

Appendix  B  also  contains  EQ  scale  intercorrelations 
and  factor  analysis  output.  Principal  axis  factor  analysis 
was  used,  and  the  2-,  3-,  and  4-factor  solutions  were 
examined,  with  solutions  rotated  to  an  oblimin  crite¬ 
rion.  As  can  be  seen  in  Appendix  B,  there  is  a  large 
positive  manifold.  Consequently,  there  is  a  large  general 
factor,  and  it  is  most  likely  that  any  other  factors  that 
emerge  will  be  at  least  moderately  correlated. 

In  the  2-factor  solution,  the  two  factors  correlate  .75. 
Factor  1  consists  of  Decisiveness,  Concentration,  Self- 
Confidence,  Task  Closure/Thoroughness,  Taking 
Charge,  Execution,  Composure,  Tolerance  for  High 
Intensity,  Sustained  Attention,  and  Flexibility.  Factor  2 
consists  of  Interpersonal  Tolerance,  Working  Coopera¬ 
tively,  Consistency  of  Work  Behaviors,  Self-Awareness, 
and  Self-Monitoring/Evaluating.  Although  the  high 
correlation  between  these  two  factors  indicates  that  a 
general  factor  accounts  for  much  of  the  variance  in  these 
two  factors,  there  is  some  unique  variance.  Factor  1 
appears  to  reflect  a  cool,  confident,  decisive  character; 
Factor  2  appears  to  reflect  a  character  that  is  self-aware 
and  works  well  with  others. 

In  the  3-factor  solution,  the  third  factor  does  not 
appear  to  add  any  useful  information.  The  4-factor 
solution  appears  to  be  interpretable.  In  this  solution,  the 
factors  are  comprised  of  the  following  scales: 


Factor  1:  Concentration,  Tolerance  for  High  Intensity, 
Composure,  Decisiveness,  Sustained 
Attention,  and  Flexibility. 

Factor  2:  Consistency  ofWork  Behaviors,  Interpersonal 
Tolerance,  and  Self-  Awareness. 

Factor  3:  Self-Monitoring/Evaluating  and  Working 
Cooperatively. 

Factor  4:  Taking  Charge,  Self-Confidence,  Task 
Closure/Thoroughness,  and  Execution. 

In  the  4-factor  solution,  the  first  factor  of  the  2-factor 
solution  is  split  into  two  parts.  One  part  (Factor  1) 
contains  scales  related  to  maintaining  attentional  focus 
and  the  ability  to  remain  composed  and  flexible.  The 
other  part  (Factor  4)  contains  scales  related  to  taking 
charge  of  situations  and  following  through.  The  second 
factor  in  the  2-factor  solution  also  split  into  two  parts  in 
the  4-factor  solution,  although  not  quite  so  tidily. 
Actually,  Working  Cooperatively  correlates  just  about 
equally  with  Factors  2  and  3  of  the  4-factor  solution.  If 
EQ  predictor  composites  were  to  be  created  at  this 
point,  the  tendency  would  be  toward  three  composites, 
drawn  from  the  4-factor  solution:  Factor  1 ,  Factor  4, 
and  the  combination  of  Factors  2  and  3. 

Summary  and  Recommendations 

The  EQ  results  in  the  pilot  test  were  promising.  Most 
of  the  scales  looked  good  in  terms  of  their  means, 
variances,  and  reliabilities.  The  two  scales  that  were 
weakest,  psychometrically,  were  Self-Awareness  and 
Self-Monitoring/Evaluating. 

Item  analysis  suggested  that  items  21,  53,  and  163 
should  be  deleted,  and  item  1 44  moved  to  a  different 
scale.  If  the  EQ  must  be  shortened,  deletion  of  scales 
rather  than  individual  items  seemed  preferable,  given 
the  high  correlations  between  scales.  However,  even  the 
scales  most  highly  correlated  (e.g..  Decisiveness  and 
Sustained  Attention,  r  =  .80,  and  Decisiveness  and 
Composure,  r  =  .81)  appear  to  be  measuring  somewhat 
different  constructs.  Based  on  considerations  including 
the  desired  length  of  the  beta  version  of  the  AT-SAT  test 
battery,  a  final  decision  was  made  to  decrease  the  EQto 
175  items.  The  Self-Monitoring  scale  was  deleted  in  its 
entirety,  and  several  scales  were  shortened  slightly. 

The  issue  of  how  to  use  the  Unlikely  Virtues  scale 
remained  unresolved.  Although  the  mean  and  standard 
deviation  for  this  scale  appeared  just  as  they  should  in 
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the  pilot  test,  this  sample  did  not  provide  any  informa¬ 
tion  about  how  much  “faking  good”  would  actually 
occur  in  an  applicant  population. 

Air  Traffic  Scenarios 

The  Air  Traffic  (AT)  Scenarios  test  consisted  of  two 
brief  practice  scenarios  of  4  to  5  minutes  each,  and  four 
test  scenarios  of  15  to  20  minutes  each.  One-fourth  of 
the  examinees  that  completed  the  AT  test  were  also 
given  a  seventh  (retest)  test  scenario  at  the  end  of  the  day. 
Two  types  of  scores  were  recorded  for  each  trial.  First, 
there  were  counts  of  different  types  of  errors,  including 
crashes  and  separation  errors  (plane-to-plane  and  plane- 
to-boundary)  and  procedural  errors  (wrong  destination, 
landing/exit  speed,  or  landing/exit  altitude).  Second, 
there  were  efficiency  measures  expressed  in  terms  of 
percentage  of  aircraft  reaching  their  target  destination 
and  delays  in  getting  the  aircraft  to  their  destination  and 
in  accepting  handoffs. 

Scoring 

Initial  inspection  of  the  results  suggested  that  crashes 
and  separation  errors  (safety)  were  relatively  distinct 
from  (uncorrelated  with)  procedural  errors.  Conse¬ 
quently,  four  separate  scores  were  generated  to  account 
for  the  data.  Initial  scores  were: 

CRASHSEP  =  crashes  +  separation  errors 
PROCERR  =  total  number  of  procedural  errors  of  all  kinds 
PCTDEST  =  percent  reaching  target  destination 
TOTDELAY  =  total  delay  (handoff  and  enroute) 

In  computing  safety  errors,  crashes  were  initially 
given  a  weight  of  4.0  to  equalize  the  variance  of  crashes 
and  separation  errors.  Since  crashes  are  relatively  rare 
events,  overweighting  crashes  led  to  reduced  consis¬ 
tency  across  trials  (reliability).  Alternative  weightings 
might  be  explored  at  a  later  date,  but  would  be  expected 
to  make  little  difference.  Consequently,  it  was  decided 
to  simply  count  crashes  as  an  additional  separation 
error. 

One  other  note  about  the  initial  computation  of 
scores  is  that  airport  flyovers  were  initially  listed  with 
separation  errors  but  appeared  to  behave  more  like 
procedural  errors.  Examinees  are  not  given  the  same 
type  of  warning  signal  when  an  aircraft  approaches  an 
airport  as  when  it  approaches  another  plane  or  a  bound¬ 
ary,  so  avoiding  airport  flyovers  was  more  a  matter  of 
knowing  and  following  the  rules. 


For  all  measures  except  PCTDEST,  the  next  step  was 
to  define  a  new  scaling  of  each  of  these  vatiables  so  that 
higher  scores  indicated  better  performance  and  so  that 
the  scale  would  be  most  sensitive  to  differences  at  higher 
levels  of  performance.  In  the  initial  scaling,  the  differ¬ 
ence  between  0  and  1  errot  was  treated  the  same  as  the 
difference  between  50  and  51  errors,  even  though  the 
former  is  a  much  more  important  distinction.  The 
transfotmations  used  were  of  the  form: 

New  Scale  =  1  /  (  a  +  b*Old  Scale) 

where  a  and  b  were  chosen  so  that  optimal  perfor¬ 
mance  would  be  around  100  and  performance  at  the 
average  of  the  old  scale  would  map  onto  50.  For  the  AT 
Test,  optimal  performance  was  indicated  by  0  on  each 
of  the  original  measures  so  that  the  transformation 
could  be  rewritten  as: 

New  Scale  =  100  /  (1  +  Old  Scale/Old  Mean). 

It  was  also  decided  to  scale  each  trial  separately.  The 
last  two  trials  were  considerably  more  difficult  than  the 
preceding  ones,  so  variance  in  performance  was  much 
higher  for  these  trials.  If  the  data  from  each  trial  were  not 
rescaled  separately,  the  last  trials  would  receive  most  of 
the  effective  weight  when  averages  were  computed. 
Consequently,  the  means  referred  to  in  the  above  for¬ 
mula  were  trial-specific  means.  The  new  scale  variables 
for  each  trial  had  roughly  equivalent  means  and  variances 
which  facilitated  cross-trial  comparisons  and  averaging. 

Case  Elimination 

During  the  initial  analyses,  prior  to  rescaling,  there 
were  several  cases  with  very  high  error  rates  or  long  delay 
times  that  appeared  to  be  outliers.  The  concern  was  that 
these  individuals  did  not  understand  the  instructions 
and  so  were  not  responding  appropriately.  (In  one  case, 
it  was  suspected  that  the  examinee  was  crashing  planes 
on  purpose.)  The  rescaling,  however,  shrunk  the  high 
end  (high  errors  or  long  times)  of  the  original  scales 
relative  to  the  lower  end,  and  after  rescaling  these  cases 
were  not  clearly  identifiable  as  outliers.  Inspection  of  the 
data  revealed  that  all  of  the  cases  of  exceptionally  poor 
performance  occurred  on  the  last  test  trial.  The  fact  that 
the  last  trial  was  exceptionally  difficult  and  that  similar 
problems  were  not  noted  on  the  earlier  trials,  suggested 
that  most  of  these  apparent  outliers  were  simply  in¬ 
stances  of  low  ability  and  not  random  or  inappropriate 
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responding.  In  the  end,  cases  with  more  than  125  crash/ 
separation  errors  or  more  than  400  minutes  of  total 
delay  time  were  flagged  as  probable  “random”  (inappro¬ 
priate)  responders.  A  total  of  16  cases  were  so  flagged. 

There  were  a  number  of  instances  of  incomplete  data. 
The  alpha  pilot  version  of  the  software  was  not  com¬ 
pletely  shock-proofed,  and  some  examinees  managed  to 
“skip  out”  of  a  trial  without  completing  it.  This  rarely 
happened  on  either  the  first  or  the  last  (fourth  test)  trial. 
Where  there  was  only  one  missing  trial,  averages  were 
computed  across  the  remaining  trials.  Where  more  than 
one  trial  was  missing,  the  overall  scores  were  set  to 
missing  as  well.  In  the  end,  we  also  flawed  cases  missing 
either  of  the  last  two  test  trials.  A  total  of  38  cases  were  so 
figged,  leaving  386  cases  vdth  no  flags  for  use  in  analyses. 

Reliability 

After  revised  scale  scores  were  computed  for  each 
trial,  reliability  analyses  were  performed.  In  this  case,  an 
ANOVA  (generalizability)  model  was  used  to  examine 
the  variance  in  scores  across  trials,  examinee  groups  (test 
orders),  and  examinees  (nested  within  groups).  The 
analyses  were  conducted  for  varying  numbers  of  trials, 
from  all  six  (two  practice  and  four  test)  down  to  the  last 
two  (test)  trials.  Table  3.3.29  shows  variance  compo¬ 
nent  estimates  for  each  of  the  sources  of  variation. 
Notwithstanding  modest  efforts  to  standardize  across 
trials,  there  was  still  significant  variation  due  to  Trial 
main  effects  in  many  cases.  These  were  ignored  in 
computing  reliabilities  (using  relative  rather  than  abso¬ 
lute  measures  of  reliability)  since  the  trials  would  be 
constant  for  all  examinees  and  would  not  contribute  to 
individual  variation  in  total  scores.  Similarly,  Group 
and  Group  by  Trial  effects  were  minimal  and  were  not 
included  in  the  error  term  used  for  computing 
reliabilities.  Group  effects  are  associated  with  different 
positions  in  the  overall  battery.  There  will  be  no  varia¬ 
tion  of  test  position  in  the  final  version  of  the  battery. 

Single  trial  reliabilities  were  computed  as  the  ratio  of 
the  valid  variance  due  to  subjects  nested  within  groups, 
SSN(Group)  to  the  total  variance,  expressed  as  the  sum 
of  SSN(Group)  and  SSN*Trial.  For  each  variable,  the 
single  trial  reliability  based  on  the  last  two  trials  was 
identical  to  the  correlation  between  the  scores  for  those 
two  trials.  Reliabilities  for  means  across  higher  num¬ 
bers  of  trials  were  computed  by  dividing  the  SSN*T 
error  component  by  the  number  of  trials.  This  is 
exactly  the  Spearman-Brown  adjustment  expressed 
in  generalizability  terms. 


Another  measure  of  reliability  was  the  correlation 
between  the  overall  scores  generated  during  the  regular 
testing  and  the  “retest”  scores  for  those  examinees  who 
completed  an  additional  trial  at  the  end  of  the  day.  T  able 
3.3.30  shows  the  correlation  between  alternative  com¬ 
posite  scores  and  the  retest  score.  The  alternative  com¬ 
posites  included  means  across  trials,  possibly  leaving  out 
the  first  few  trials,  and  a  weighted  composite  giving 
increasing  weight  to  the  later  composites.  (For  AT,  the 
weights  were  0  for  the  practice  trials  and  1,2,3,  and  4 
for  the  test  trials.  For  the  TW  test,  the  weights  were  1 , 
2,  and  3  for  the  three  regular  trials.)  The  row  labeled 
SEPSKl-6,  for  example,  corresponds  to  the  simple 
mean  across  all  six  (two  practice  and  four  test)  trials. 
Since  the  retest  was  a  single  trial  and,  in  most  cases,  the 
composite  score  from  regular  testing  encompassed  more 
than  one  trial,  the  two  measures  being  correlated  do  not 
have  equal  reliability.  In  general,  as  expected,  the  corre¬ 
lations  ranged  between  the  values  estimated  for  single 
trial  reliabilities  and  reliability  estimates  based  on  the 
number  of  trials  included  in  the  composite  scores.  In 
some  cases,  these  “test-retest”  reliabilities  were  lower 
than  the  internal  consistency  estimates,  indicating  some 
individual  differences  in  the  retention  of  skill  over  the 
course  of  the  testing  day. 

Based  on  analyses  of  the  reliability  data,  it  was 
concluded  that  the  most  appropriate  scores  for  use  with 
the  pilot  data  were  averages  of  the  last  two  test  trials.  The 
2-trial  reliability  for  these  scores  was  higher  than  the 
three-trial  reliability  for  the  last  3  scores,  the  4-trial 
reliability  for  the  last  four  scores,  and  so  on.  The 
composite  based  on  the  last  two  trials  also  had  the 
highest  correlation  with  the  retest  scores  in  most  cases  or 
was  at  least  a  close  second. 

Summary  and  Recommendations 

It  was  felt  if  separate  scores  were  to  be  used  in  the 
concurrent  validation,  additional  practice  and  test  trials 
would  be  needed  to  achieve  a  high  level  of  reliability  for 
the  “Separation  Skill”  variable.  It  was  recommended 
that  three  practice  trials  be  used  with  each  trial  targeted 
to  test  understanding  of  specific  rules  and  more  tai¬ 
lored  feedback  after  each  trial.  For  example,  the  first 
trial  might  include  four  planes,  two  headed  for  their 
designated  airport  runways  and  two  headed  for  their 
designated  exit  gates.  One  of  the  two  exiting  planes 
would  be  at  the  wrong  level  and  the  other  at  the  wrong 
speed.  Similarly,  one  of  the  landing  planes  would  be  at 
the  wrong  level  and  the  other  at  the  wrong  speed.  No 
changes  in  direction  would  be  required.  At  the  end  of  a 
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very  brief  time,  it  could  be  determined  whether  the 
examinee  changed  level  and  speed  appropriately  for 
each  aircraft,  with  feedback  if  they  did  not. 

A  second  example  might  involve  turning  planes  to  get 
to  their  destinations.  Specific  feedback  on  changing 
directions  would  be  given  if  the  planes  failed  to  reach 
their  assigned  destination.  Further  testing  of  speed  and 
level  rules  would  also  be  included.  The  final  practice 
scenario  would  involve  airport  landing  directions  and 
flyovers. 

Following  the  three  practice  scenarios  (which  might 
take  a  total  of  10  minutes  to  run  with  another  10 
minutes  for  feedback),  five  test  scenarios  with  increasing 
difficulties  were  proposed.  The  alpha  fourth  test  sce¬ 
nario  may  be  a  bit  too  difficult  and  might  be  toned  down 
a  little.  However,  controller  performance  is  expected  to 
be  at  a  much  higher  level,  so  at  least  two  relatively 
difficult  scenarios  should  be  included.  After  three  prac¬ 
tice  and  three  easier  test  scenarios,  performance  on  the 
last  two  more  difficult  scenarios  should  be  quite  reliable. 

Software  Changes 

After  the  Alpha  Version  pilot  test,  the  AT  test  was 
changed  to  have  more  extensive  and  more  highly  edited 
instructions  and  was  converted  to  a  32-bit  version  to  run 
under  Windows  95.  The  practice  scenarios  were  modi¬ 
fied  to  “teach”  specific  aspects  of  the  exercises  (changing 
speed  and  level  in  practice  1,  changing  directions  in 
practice  2,  noticing  airport  landing  directions,  and 
coping  with  pilot  readback  errors  in  practice  3).  Specific 
feedback  was  provided  after  each  practice  session  keyed  to 
aspects  of  the  examinee’s  performance  on  the  practice  trial. 

The  '  ‘new”  version  of  the  scenario  player  provided 
slightly  different  score  information.  In  particular  the 
“en  route  delay”  variable  was  computed  as  the  total  en 
route  time  for  planes  that  landed  correctly.  We  modi¬ 
fied  the  shell  program  to  read  the  “replay”  file  and  copy 
information  from  the  “exit”  records  (type  XT)  into  the 
examinee’s  data  file.  This  allowed  us  to  record  which 
planes  either  crashed  or  were  still  flying  at  the  end  of  the 
scenario.  We  computed  a  “total  en  route”  time  to 
replace  the  “delay”  time  provided  by  the  Alpha  version. 

Time  Wall/Pattern  Recognition  Test 

The  analyses  for  the  Time  Wall  (TW)  test  were  very 
similar  to  the  analyses  performed  for  the  Air  Traffic 
Scenarios  test.  One  difference  was  that  TW  had  three 
exactly  parallel  trials  instead  of  two  practice  and  four  test 
scenarios  that  differed  in  difficulty.  Each  TW  trial  had 
a  brief  “practice”  trial  where  no  results  were  recorded. 


The  three  scores  analyzed  for  the  TW  test  were  (a) 
Pattern  Recognition  Accuracy  (PRACCY),  defined  as 
the  percent  of  correct  pattern  matching  responses  out  of 
all  correct  and  incorrect  responses  (e.g.,  excluding  time¬ 
outs);  (b)  Pattern  Recognition  Speed  (PRSPD),  a  trans¬ 
formation  of  the  average  time,  in  milliseconds,  for 
correct  responses;  and  (c)  Time  Wall  Accuracy 
(TWACCY),  a  transformation  of  the  mean  absolute 
time  error,  in  milliseconds.  The  transformations  used 
for  Pattern  Recognition  Speed  and  Time  Wall  Accuracy 
were  identical  in  form  to  those  used  with  the  AT  test.  In 
this  case,  however,  the  transformations  mapped  the 
maximum  value  to  about  100  and  the  mean  value  to 
about  50  across  all  trials,  rather  than  using  a  separate 
transformation  for  each  trial.  This  was  done  because  the 
trials  did  not  vary  in  difficulty  for  TW  as  they  did  for  AT. 

Case  Elimination 

Figure  3.3. 1  shows  a  plot  of  the  Pattern  Recognition 
Accuracy  and  Speed  variables.  A  number  of  cases  had 
relatively  high  speed  scores  and  lower  than  chance 
(50%)  accuracy  scores.  In  subsequent  analyses,  all  cases 
with  an  accuracy  score  less  than  40  on  any  of  the 
operational  trials  were  deleted.  This  resulted  in  a  dele¬ 
tion  of  12  participants. 

Reliability 

Tables  3.3.29  and  3.3.30  show  internal  consistency 
and  test-retest  reliability  estimates  for  TW  as  well  as  for 
AT.  Analyses  of  these  data  suggested  that  averaging 
across  all  three  trials  led  to  the  most  reliable  composite 
for  use  in  analyses  of  the  pilot  data. 

Summary  and  Recommendations 

Time  Wall  Accuracy  reliability  estimates  were  modest, 
although  the  test-retest  correlations  held  up  fairly  well. 
Preliminary  results  suggested  that  five  or  six  trials  may  be 
needed  to  get  highly  reliable  results  on  all  three  measures. 

Software  Changes 

The  trial  administration  program  was  changed  to  allow 
us  to  specify  the  number  of  Time  Wall  items  administered 
and  to  shut  off  the  “warm  up”  trials  for  each  administration. 
The  main  program  then  called  the  trial  administration 
program  6  times.  The  first  three  trials  had  5  Time  Wall 
items  each  and  were  considered  test  trials.  The  next  three 
trials  had  25  Time  Wall  items  each  and  were  considered  test 
trials.  After  the  practice  trials,  the  examinee’s  performance 
was  analyzed  and  specific  feedback  was  given  on  how  to 
improve  their  score. 
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Testing  Times 

Table  3.3.31  shows  distributional  statistics  for  in¬ 
struction  time  and  total  time  for  the  AT  and  TW  tests 
in  their  current  form.  While  there  was  some  variation  in 
instruction  time,  the  total  times  were  quite  close  to  the 
original  targets  (90  and  25  minutes,  respectively). 

Conclusions 

The  purpose  of  the  pilot  study  was  to  determine  if  the 
predictor  battery  required  revisions  prior  to  its  use  in  the 
proposed  concurrent  validation  study.  A  thorough  analy¬ 
sis  of  the  various  tests  was  performed.  A  number  of 
recommendations  related  to  software  presentation  - 
item  changes,  and  predictor  construct  revisions  -  were 
outcomes  of  the  pilot  study.  The  project  team  believed 
that  the  changes  made  to  the  test  battery  represented  a 
substantial  improvement  over  initial  test  development. 
The  beta  battery,  used  in  the  concurrent  validation 
study,  was  a  professionally  developed  set  of  tests  that 
benefited  greatly  from  the  pilot  study. 
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Angles 

The  Angles  test  measures  the  participant’s  ability  to  recognize  angles.  This  test  contains  30 
multiple-choice  questions  and  allows  participants  up  to  8  minutes  to  complete  them.  The 
score  is  based  on  the  number  of  correct  answers  (with  no  penalty  for  wrong  or  unanswered 
questions).  There  are  two  types  of  questions  on  the  test.  The  first  presents  a  picture  of  an 
angle  and  the  participant  chooses  the  correct  answer  of  the  angle  (in  degrees)  from  among  4 
response  options.  The  second  presents  a  measure  in  degrees  and  the  participant  chooses  the 
angle  (among  4  response  options)  that  represents  that  measme. 

For  each  worker  requirement  listed  below,  enter  the  rating  best  describing  the  extent  to  which 
this  test  and/or  its  subtests  measure  that  particular  worker  requirement. 

5=This  test  measures  this  worker  requirement  to  a  very  great  extent 
4=This  test  measures  this  worker  requirement  to  a  considerable  extent 
3=This  test  measures  this  worker  requirement  to  a  moderate  extent 
2=This  test  measures  this  worker  requirement  to  a  limited  extent 
l=This  test  measures  this  worker  requirement  to  a  slight  extent 

_ 0=This  test  does  not  measure  this  worker  requirement _ 

Figure  2.1.  Sample  Description  of  an  AT-SAT  Measure. 
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Figure  2.2.  Example  of  Linkage  Rating  Scale. 


FIGURES  AND  TABLES 


PRACCY  j 
100  + 

90  + 


80  + 


70  + 


60  + 


SO  -f 


40 


112  1211  2131333 
3  131211111211232  234 

11  21111111122113212 
312  2111111111112122123  3 

I  11  111111111111122  3 

2  21  1  1111111111121122323 
2  1111111111111123  1 

1  1  1114111111111112212  4 

II  1  11121111211213  2 

1  1  11211111112231 

221  13  1121111311113  2 

1  1311111  121  34223  3 

111  121211121  3 
1  21  1  11313312322 

1  1 


1323 

233 

1  2 

1 

13  134 

23 

13  3 

21 

1  34 

121 

1444 

3  4  3 

2 

2  1 

13  2 

1  1 

4  2 

3  22 

3 

1 

11 

22 

312  2 

244  3 

1 

2  2 

22 

413  3 

4  33 

12 

3  3 

2  3  2  4 

3  14  4 

1 

1 

2 

1  32  4113  334 

3 

2 

2 

23  32 

3 

1 

2 

3  1 

3  1 

2  234 

3  3 

2  3 

3 

2 

1 

1  1 

3 

30  'f 


OUTLIERS 


20  -f 


10  + 


1 

1 

1  12  3 

4 

4  3  2 

2  4 


0  + 


+ 


+ 


0 


20 


40 


60 


80 


100 


-+- 

120 


PRSPEED 

MOTE:  39  obs  had  missincr  values.  999  obs  hidden. 


Figure  3.3.1.  Plot  of  PRACC  Y*PRSPEED.  Symbol  is  value  of  TRIAL 
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Table  2.1.  SACHA-Generated  Worker  Requirements 


COGNITIVE  ABILITIES 

REASONING 

1. 

Deductive  Reasoning 

2. 

Inductive  Reasoning 

3. 

Chunking 

4. 

Mathematical  Reasoning 

5. 

Visuospatial  Reasoning 

6. 

Mechanical  Reasoning 

7. 

Verbal  Reasoning 

COMPUTATIONAL  ABILITY 

8. 

Number  Facility 

9. 

Geometry 

COMMUNICATION 

10. 

Oral  Communication 

11. 

Listening 

12. 

Reading 

13. 

Writing 

ATTENTION 

14. 

Selective  Attention 

15. 

Time  Sharing 

16. 

Vigilance 

MEMORY 

17. 

Short-Term  Memory 

18. 

Intermediate-Term  Memory 

19. 

Long-Term  Memory 

20. 

Associative  Memory 

21. 

Visual  Memory 

22. 

Auditory  Memory 

23. 

Numerical  Memory 

METACOGNITIVE 

Local 

24. 

Thinking  Ahead 

25. 

Problem  Recognition  and  Definition 

26. 

Deciding  on  the  Steps  to  solve  the  Problem 

27. 

Ordering  the  Steps 

28. 

Creating  a  Mental  Representation  for  Information 

29. 

Monitoring  Performance 

30. 

Evaluating  Performance 

Global 

31. 

Prioritization 

32. 

Multitasking 

(Continued) 
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Table  2.1.  SACHA-Generated  Worker  Requirements  (continued) 


INFORMATION  PROCESSING 

33.  Encoding 

34.  Rule  Inference 

35.  Rule  Application 

36.  Confirmation 

37.  Sequential  Distribution  of  Processing  Steps 

38.  Information  Processing  Strategies 

39.  Learning 

40.  Flexibility 

PERCEPTUAL/SPATIAL  ABILITIES 

PERCEPTUAL  ABILITIES 

41. 

Dynamic  Visual-Spatial  Ability 

42. 

Movement  Detection 

43. 

Verbal  and  Figural  Closure 

44. 

Perceptual  Speed  and  Accuracy 

SPATIAL  ABILITIES 

45. 

Projection 

Flexibility  of  Closure 

47. 

Scanning  Efficiency 

48. 

Two-Dimensional  Mental  Rotation 

49. 

Three-Dimensional  Mental  Rotation 

50. 

Spatial  Scanning 

TEMPERAMENT/INTERPERSONAL 

INTERPERSONAL 

51. 

Working  Cooperatively 

52. 

Establishing  Rapport 

WORK  &  EFFORT 

53. 

Achievement/Self-Confidence 

54. 

General  Activity 

55. 

Concentration 

56. 

Taking  Charge 

57. 

Adherence  to  a  Work  Ethic 

58. 

Thoroughness  and  Attentiveness  to  Details 

59. 

Thinking  Ahead 

60. 

Decisiveness 

61. 

Realistic  Orientation 

STABILITY/ADJUSTMENT 

62. 

Stress  Tolerance 

63. 

Flexibility 

SELF-EFFICACY 

64. 

Internal  Locus  of  Control 

65. 

Self-Sufficiency 

PSYCHOMOTOR  ABILITIES 

66. 

Multilimb  Coordination 

67. 

Control  Precision 

68. 

Reaction  Time 

69. 

Response  Orientation 

70. 

Finger  Dexterity 

71. 

Manual  Dexterity 

72. 

Eye-Hand  Coordination 

73. 

Response  Integration 
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Table  2.2.  Worker  Requirements  Generated  by  Subject  Matter  Experts 


COGNITIVE 

1. 

Oral  Communication 

2. 

Written  Communication 

3. 

Reading 

4. 

Active  Listening 

5. 

Translating  Information 

6. 

Interpreting  Information 

7.  ■ 

Summarizing  Information 

8. 

Numeric  Ability  (addition/subtraction) 

9. 

Numeric  Ability  (multiplication/division) 

10. 

“Street”  Physics 

11. 

Short-Term  Memory 

12. 

Intermediate-Term  Memory 

13. 

Long-Term  Memory 

14. 

Thinking  Ahead 

15. 

Planning 

16. 

Prioritization 

17. 

Execution 

18. 

Problem  Solving 

19. 

Reasoning  (Thinking) 

20. 

Creativity 

21. 

Self-Monitoring/Evaluating 

22. 

Rule  Application 

23. 

Sustained  Attention 

24. 

Time  Sharing 

25. 

Aviation  Science  Background 

26. 

Geography  . 

SPATIAL 

-  27.' 

Situational  Awareness  •  •- 

28. 

Visualization 

29. 

Projection 

Scanning 

INTERPERSONAL 

31. 

Professionalism 

32. 

Working  Cooperatively 

33. 

Personal  Tolerance 

WORK/EFFORT 

34. 

Self-Esteem 

35. 

Self-Confidence 

36. 

Aggressiveness  _ 

37. 

Self-Awareness 

.38. 

.  Attention  to  Detail  „  . . 

39. 

Task  Closure/Thoroughness 

40. 

Decisiveness 

41. 

Consistency 

42. 

Flexibility 

43. 

Concentration 

44. 

Composure 

45. 

Tolerance  for  High-Intensity  Work 

46. 

Motivation 

47. 

Commitment  to  Job 

65 


FIGURES  AND  TABLES 


Table  2.3.  Revised  Consolidated  Worker  Requirements  List,  With  Definitions 


SME-Generated  WORKER  REQUIREMENTS 


Oral  Communication 

The  ability  to  speak  clearly  and  concisely  to  individuals  so  they  understand  what 
is  being  communicated.  Projecting  a  confident  tone  of  voice  is  an  important 
component  of  this  ability. 

Written  Communication 

The  ability  to  write  legibly  and  accurately  (e.g.,  strip  markings). 

Reading 

The  ability  to  read  and  understand  written  information  (e.g.,  ATCS  documents, 
manuals). 

Active  Listening 

The  ability  to  hear  and  comprehend  spoken  information.  This  ability  requires  an 
individual  to  recognize  or  pick  out  pertinent  auditory  information. 

Translating  Information 

The  ability  to  translate  symbols  or  symbolic  abbreviations  into  meaningful 
information. 

Interpreting  Information 

The  ability  to  put  information  into  meaningful  tenns.  It  is  the  ability  to 
recognize  the  implications  for  a  statement  or  condition  (e.g.,  cold  front). 

Summarizing  Information 

The  ability  to  summarize  and  consolidate  information  most  relevant  to  the 
situation. 

Numeric  Ability  (add/sub) 

The  ability  to  quickly  and  accurately  perform  basic  math  operations  (addition 
and  subtraction). 

Numeric  Ability  (mult/div) 

The  ability  to  quickly  and  accurately  perform  basic  math  operations 
(multiplication  and  division). 

Street  Physics 

The  ability  to  understand  the  general  rules  of  science  as  practiced  in  the  ATCS 
environment  (e.g.,  aircraft  size  and  turbulence  effects,  angles  of  intersect, 
overtake  speed,  headings). 

Short-Term  Memory 

The  ability  to  remember  pertinent  information  within  a  brief  period  of  time  (less 
than  1  minute).  Examples  of  information  include  call  signs  and  keywords. 

Intermediate-Term  Memory 

The  ability  to  remember  pertinent  information  over  a  1-10  minute  period. 

Long-Term  Memory 

The  ability  to  remember  pertinent  information  over  long  periods  of  time. 

Examples  of  information  include  maps  and  separation  procedures. 

Thinking  Ahead 

The  ability  to  anticipate  or  recognize  problems  before  they  occur  and  to  develop 
plans  to  avoid  problems.  This  includes  thinking  about  what  might  happen. 

Planning 

The  ability  to  determine  the  appropriate  course(s)  of  action  to  take  in  any  given 
situation. 

Prioritization 

The  ability  to  identify  activities  that  are  most  critical  and  require  immediate 
attention.  This  involves  a  constant  evaluation  of  new  information  followed  by 
re-prioritization  of  job  activities. 

Execution 

The  abilitv  to  take  timelv  action  in  order  to  avoid  problems  and  to  solve  existing 
problems. 

Problem  Solving 

The  ability  to  identify  a  potential  problem  or  existing  problem  and  to  identify 
the  variables  used  in  solving  the  problem. 

Reasoning 

The  ability  to  apply  available  information  to  make  decisions,  draw  conclusions, 
or  identify  alternative  solutions. 

Creativity 

The  ability  to  identify  new  or  novel  solutions  to  potential  problems  when 
existing  or  established  solutions  no  longer  applies. 

Self-Monitoring/  Evaluation 

The  ability  and  willingness  to  check  your  own  work  performance,  evaluate  the 
effectiveness  of  your  decisions,  and  alter  your  performance  if  necessary. 

Rule  Application 

The  ability  to  apply  learned  rules  to  the  real  work  situation 

(Continued) 
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Table  2.3.  Revised  Consolidated  Worker  Requirements  List,  With  Definitions  (Continued) 


Sustained  Attention 

The  ability  to  stay  focused  on  a  task(s)  for  long  periods  of  time  (over  60 
minutes). 

Time  Sharing 

The  ability  to  perform  two  or  more  job  activities  at  the  same  time. 

Situational  Awareness 

Being  cognizant  of  all  information  within  a  four-dimensional  space  (i.e., 
separation  standards  plus  time).  This  involves  the  ability  to  “understand”  the 
airspace  as  an  integrated  whole  (e.g.,  getting  the  picture). 

Visualization 

The  ability  to  translate  material  into  a  visual  representation  of  what  is  currently 
occurring. 

Projection 

The  ability  to  translate  material  into  visual  representation  of  what  will  occur  in 
the  future. 

Scanning 

The  ability  to  quickly  and  accurately  search  for  information  on  a  computer 
screen,  radar  scope,  or  computer  printout. 

Professionalism 

The  ability  to  establish  respect  and  confidence  in  your  abilities  among  other 
controllers. 

Working  Cooperatively 

The  willingness  to  work  with  others  to  achieve  a  common  goal.  This  includes  a 
willingness  to  voluntarily  assist  another  controller  if  the  situation  warrants. 

Interpersonal  Tolerance 

The  ability  to  accommodate  or  deal  with  differences  in  personalities,  criticisms, 
and  interpersonal  conflicts  in  the  work  environment. 

Self-Esteem 

Having  a  positive  opinion/image  of  oneself. 

Self-Confidence 

A  belief  that  you  are  the  person  for  the  job  and  knowing  that  your  processes  and 
decisions  are  correct. 

Taking  Charge 

The  ability  to  take  control  of  a  situation  and  reach  out  and  take  correct  action. 

Self-Awareness 

The  ability  to  maintain  an  internal  awareness  of  your  actions  and  attitudes.  This 
includes  knowing  your  limitations. 

Attention  to  Detail 

The  ability  to  recognize  and  attend  to  the  details  of  the  job  that  others  might 
overlook. 

Task  Closure/  Thoroughness 

The  ability  to  continue  an  activity  to  completion  through  the  coordination  and 
inspection  of  work. 

Decisiveness 

The  ability  to  make  effective  decisions  in  a  timely  manner. 

Behavioral  Consistency 

The  ability  to  behave  consistently  at  work  (e.g.,  dealing  with  coworkers  in  a 
consistent  manner;  consistently  using  the  correct  phraseology). 

Flexibility  (Stability/ 
Adjustment) 

The  ability  to  adapt  to  changing  situations  or  conditions. 

Concentration 

The  ability  to  focus  on  job  activities  amid  distractions  for  short  periods  of  time. 

Composure 

The  ability  to  think  clearly  in  stressful  situations. 

Tolerance  for  High  Intensity 

The  ability  to  perform  effectively  and  think  clearly  during  heavy  work  flow. 

Motivation 

The  desire  to  motivate  oneself  through  challenges  on  the  job  and  to  progress  to  a 
higher  level  of  skill. 

Commitment  to  the  Job 

The  desire  to  be  an  ATCS  and  work  hard  to  be  successful. 

(Continued) 
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Table  2.3.  Revised  Consolidated  Worker  Requirements  List,  With  Definitions  (Continued) 


SACHA-Generated  WORKER  REQUIREMENTS 


Confirmation 

The  ability  to  efficiently  select  a  response  option  consistent  with  the  application 
of  inferred  rules. 

Rule  Application 

The  ability  to  efficiently  apply  transformational  rules  inferred  from  the  complete 
portions  of  the  stimulus  array  to  the  incomplete  portion  of  the  array. 

Perceptual  Speed  and 
Accuracy 

Ability  to  perceive  visual  information  quickly  and  accurately  and  to  perform 
simple  processing  tasks  with  (e.g.,  comparisons). 

Internal  Locus  of  Control 

Believes  that  individuals  have  influence  over  the  outcome  of  an  event;  takes 
responsibility  for  outcomes. 

Visuospatial  Reasoning 

Ability  to  perceive  and  understand  principles  governing  relationships  among 
several  figures. 

Flexibility  (Information 
Processing) 

The  ability  to  find  new  meanings  for  stimuli,  to  combine  stimulus  attributes  to 
come  up  with  new  and  different  solution  protocols,  and  to  employ  flexible  ways 
of  relating  new  information  to  stored  knowledge. 

Dynamic  Visual-Spatial 

Ability  to  deal  with  dynamic  visual  movement. 

Verbal  Reasoning 

The  ability  to  perceive  and  understand  principles  governing  the  use  of  verbal 
concepts  and  symbols. 

Learning 

Changes  in  information-processing  strategies  over  time  or  trials  not  due  to 
maturation  or  aging. 

Encoding 

Transformation  or  translation  of  information;  coding;  decoding. 

Movement  Detection 

Ability  to  detect  physical  movement  of  objects  and  to  Judge  their  direction. 

Chunking 

The  ability  to  organize  stimuli  into  meaningful  groups  or  units. 

Mathematical  Reasoning 

Ability  to  perceive  and  understand  principles  governing  the  use  of  quantitative 
concepts  and  symbols. 

Mechanical  Reasoning 

Ability  to  perceive  and  understand  the  relationship  of  physical  forces  and 
mechanical  elements  in  a  prescribed  situation. 

Angles 

The  ability  to  apply  the  principles  of  geometry  to  angles  and  computations 
involving  angles.  The  ability  involves  both  the  speed  and  accuracy  of 
computation. 

Rule  Inference 

The  ability  to  efficiently  ascertain  the  rules  governing  relations  between 
stimulus  attributes. 

Realistic  Orientation 

Prefers  dealing  with  activities  which  have  tangible  and  measurable 
consequences;  enjoys  activities  which  require  skill,  is  reinforced  by 
accomplishing  realistic  tasks. 

2-D  Mental  Rotation 

Ability  to  identify  a  two-dimensional  figure  when  seen  at  different  angular 
orientations. 

3-D  Mental  Rotation 

Ability  to  identify  a  three-dimensional  object  when  seen  at  different  angular 
orientations  either  within  the  picture  plane  or  about  the  axis  in  depth. 

OTHER  WORKER  REQUIREMENTS 


Translation  of  Uncertainty 

The  ability  to  assign  a  subjective  probability  regarding  the  likelihood  of  an  event 
occurring;  the  ability  to  u.se  probabilities  to  identify  optimal  courses  of  action 
(CTA,  1988). 

Recall  from  Interruption 

The  ability  to  recall  a  deferred  or  interrupted  action  when  priorities  permit,  and 
to  be  able  to  resume  the  action  appropriately  (Ammerman  et  al.,  1983) 
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Table  2.4.  Mean  Worker  Requirement  Ratings  Rank  Ordered  for  all  ATCSs 


Learning  the  Job 

Doing  the  Job 

MEAN 

SD 

LABEL 

MEAN 

SD 

LABEL 

4.43 

0.71 

Active  Listening 

4.60 

0.62 

Tolerance  for  High  Intensity 

4.32 

0.79 

Tolerance  for  High  Intensity 

4.56 

0.63 

Oral  Communication 

4.30 

0.83 

Prioritization 

4.53 

0.66 

Active  Listening 

4.27 

0.80 

Composure 

4.53 

0.70 

Prioritization 

4.25 

0.87 

Oral  Communication 

4.50 

0.70 

Composure 

4.21 

0.84 

Planning 

4.45 

0.69 

Planning 

4.19 

0.84 

Rule  Application 

4.40 

0.71 

Decisiveness 

4.18 

0.85 

Execution 

4.37 

0.80 

Execution 

4.16 

0.85 

Short-Term  Memory 

4.35 

0.81 

Thinking  Ahead 

4.15 

0.86 

Reading 

4.33 

0.86 

Situational  Awareness 

4.15 

0.82 

Decisiveness 

4.33 

0.79 

Scanning 

4.13 

0.82 

Reasoning 

4.31 

0.74 

Reasoning 

4.12 

0.92 

Thinking  Ahead 

4.30 

0.80 

Short-Term  Memory 

4.12 

0.90 

Scanning 

4.30 

0.73 

Taking  Charge 

4.11 

0.88 

Long-Term  memory 

4.26 

0.78 

Time  Sharing 

4.08 

0.93 

Situational  Awareness 

4.26 

0.85 

Visualization 

4.06 

0.91 

Visualization 

4.26 

0.84 

Projection 

4.06 

0.83 

Concentration 

4.25 

0.79 

Rule  Application 

4.05 

0.85 

Problem  Identification 

4.23 

0.83 

Problem  Identification 

4.05 

0.91 

Projection 

4.23 

0.80 

Concentration 

4.04 

0.83 

Flexibility  (S/A) 

4.22 

0.77 

Flexibility  (S/A) 

4.02 

0.92 

Taking  Charge 

4.13 

0.83 

Perceptual  Speed  and  Accuracy 

4.01 

0.91 

Time  Sharing 

4.11 

0.92 

Long-Term  Memory 

3.99 

0.87 

Perceptual  Speed  and  Accuracy 

4.11 

0.78 

Self-Awareness 

3.98 

0.92 

Commitment  to  the  Job 

4.10 

0.81 

Working  Cooperatively 

3.91 

0.86 

Summarizing  Information 

4.07 

0.94 

Sustained  Attention 

3.90 

0.90 

Self-Awareness 

4.06 

0.83 

Summarizing  Information 

3.89 

0.93 

Self-Monitoring/Evaluating 

4.04 

0.84 

Self-Monitoring/Evaluating 

3.87 

0.92 

Translating  Information 

4.02 

0.78 

Self-Confidence 

3.87 

0.87 

Intermediate-Term  Memory 

3.96 

0.88 

Intermediate-Term  Memory 

3.87 

0.96 

Sustained  Attention 

3.96 

0.85 

Task  Closure/Thoroughness 

3.87 

0.86 

Self-Confidence 

3.95 

0.89 

Reading 

3.86 

0.87 

Task  Closure/Thoroughness 

3.92 

0.90 

Creativity 

3.82 

0.96 

Working  Cooperatively 

3.92 

0.90 

Recall  From  Interruption 
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Table  2.4.  Mean  Worker  Requirement  Ratings  Rank  Ordered  for  all  ATCSs  (Continued) 


Learning  the  Job 

Doing  the  Job 

MEAN 

SD 

LABEL 

MEAN 

SD 

LABEL 

3.82 

0.87 

Self-Esteem 

3.89 

0.86 

Verbal  Reasoning 

3.81 

0.86 

Verbal  Reasoning 

3.88 

0.98 

Translating  Information 

3.78 

0.81 

Behavioral  Consistency 

3.88 

0.92 

Commitment  to  the  Job 

3.77 

0.91 

Recall  From  Interruption 

3.83 

0.92 

Internal  Locus  of  Control 

3.74 

0.95 

Visuospatial  Reasoning 

3.80 

0.85 

Attention  to  Detail 

3.72 

0.98 

Interpreting  Information 

3.80 

1.02 

Dynamic  Visual-Spatial 

3.71 

0.95 

Internal  Locus  of  Control 

3.80 

1.03 

Movement  Detection 

3.70 

1.05 

Movement  Detection 

3.79 

0.87 

Professionalism 

3.69 

0.95 

Professionalism 

3.78 

1.00 

Interpreting  Information 

3.69 

0.99 

Dynamic  Visual-Spatial 

3.78 

0.97 

Visuospatial  Reasoning 

3.68 

0.99 

Creativity 

3.77 

0.88 

Self-Esteem 

3.68 

0.90 

Interpersonal  Tolerance 

3.76 

0.85 

Behavioral  Consistency 

3.67 

0.97 

Street  Physics 

3.74 

1.00 

Street  Physics 

3.67 

0.93 

Motivation 

3.72 

0.85 

Flexibility  (IP) 

3.59 

0.90 

Flexibility  (IP) 

3.68 

0.86 

Interpersonal  Tolerance 

3.58 

0.92 

Attention  to  Detail 

3.67 

1.01 

T ranslation  of  Uncertainty 

3.55 

0.99 

Translation  of  Uncertainty 

3.67 

0.91 

Aviation  Science  Background 

3.50 

0.93 

Chunking 

3.53 

0.92 

Chunking 

3.48 

1.02 

Aviation  Science  Background 

3.47 

0.93 

Motivation 

3.39 

0.93 

Written  Communication 

3.40 

0.96 

Written  Communication 

3.18 

1.11 

2-D  Mental  Rotation 

3.32 

1.13 

Geography 

3.18 

1.17 

3-D  Mental  Rotation 

3.24 

1.20 

3-D  Mental  Rotation 

3.16 

1.03 

Realistic  Orientation 

3.20 

1.12 

2-D  Mental  Rotation 

3.14 

1.12 

Geography 

3.19 

1.02 

Realistic  Orientation 

3.13 

0.98 

Mechanical  Reasoning 

3.15 

1.00 

Mechanical  Reasoning 

3.12 

0.99 

Mathematical  Reasoning 

3.08 

1.01 

Mathematical  Reasoning 

2.96 

0.96 

Numeric  Ability  (add/sub) 

2.91 

1.02 

Numeric  Ability  (add/sub) 

2.83 

1.12 

Angles 

2.79 

1.13 

Angles 

2.64 

1.00 

Numeric  Ability  (mult/div) 

2.63 

1.04 

Numeric  Ability  (mult/div) 
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Table  2.5.  Worker  Requirement  Ratings  for  Doing  the  Job  for  the  Three  Options  and  All  ATCSs 


WORKER  REQUIREMENT 
LABEL 

ARTCC 

TERMINAL 

FLIGHT  SERVICE 

ALL  ATCSs 

MEAN 

SD 

MEAN 

SD 

MEAN 

SD 

MEAN 

SD 

Oral  Communication 

4.56 

0.56 

4.57 

0.66 

4.63 

0.53 

4.56 

0.63 

Written  Communication 

3.18 

0.94 

3.43 

0.88 

3.49 

1.11 

3.40 

0.96 

Reading 

3.79 

0.98 

3.87 

0.89 

4.26 

0.78 

3.95 

0.89 

Active  Listening 

4.59 

0.60 

4.55 

0.65 

4.52 

0.65 

4.53 

0.66 

Translating  Information 

3.68 

1.08 

3.73 

0.98 

4.32 

0.74 

3.88 

0.98 

Interpreting  Information 

3.30 

1.02 

3.59 

0.97 

4.51 

0.64 

3.78 

1.00 

Summarizing  Information 

3.94 

0.87 

3.92 

0.86 

4.46 

0.62 

4.06 

0.83 

Numeric  Ability  (add/sub) 

3.00 

1.23 

2.88 

1.01 

3.02 

0.86 

2.91 

1.02 

Numeric  Ability  (mult/div) 

2.75 

1.28 

2.58 

1.01 

2.66 

0.95 

2.63 

1.04 

Street  Physics 

3.97 

0.98 

3.82 

3.38 

1.00 

1.00 

3.74 

1.00 

Short-Term  Memory 

4.34 

0.77 

4.35 

0.80 

4.17 

0.79 

4.30 

0.80 

Intermediate-Term  Memory 

3.94 

0.94 

3.93 

0.86 

4.03 

0.80 

3.96 

0.88 

Long-Term  Memory 

4.29 

0.80 

4.11 

0.80 

4.01 

0.91 

4.11 

0.92 

Thinking  Ahead 

4.50 

0.64 

4.54 

0.64 

3.81 

1.04 

4.35 

0.81 

Planning 

4.52 

0.59 

4.54 

0.62 

4.18 

0.86 

4.45 

0.69 

Prioritization 

4.69 

0.58 

4.61 

0.63 

4.23 

0.84 

4.53 

0.70 

Execution 

4.51 

0.61 

4.50 

0.74 

3.97 

0.94 

4.37 

0.80 

Problem  Identification 

4.31 

0.76 

4.33 

0.75 

3.96 

0.97 

4.23 

0.83 

Reasoning 

4.42 

0.70 

4.33 

0.69 

4.16 

0.86 

4.31 

0.74 

Creativity 

4.06 

0.81 

3.97 

0.88 

3.66 

0.95 

3.92 

0.90 

Self-Monitoring/Evaluating 

4.09 

0.93 

4.03 

0.83 

3.96 

0.77 

4.04 

0.84 

Rule  Application 

4.18 

0.87 

4.29 

0.75 

4.20 

0.83 

4.25 

0.79 

Sustained  Attention 

4.13 

0.86 

4.14 

0.92 

3.87 

1.01 

4.07 

0.94 

Time  Sharing 

4.41 

0.67 

4.37 

0.74 

3.91 

0.85 

4.26 

0.78 

Situational  Awareness 

4.54 

0.70 

4.46 

0.75 

3.81 

1.02 

4.33 

0.86 

Visualization 

4.21 

0.90 

4.33 

0.79 

4.12 

0.96 

4.26 

0.85 

Projection 

4.24 

0.90 

4.36 

0.78 

4.04 

0.91 

4.26 

0.84 

Scanning 

4.32 

0.75 

4.36 

0.80 

4.27 

0.80 

4.33 

0.79 

Professionalism 

3.82 

0.92 

3.79 

0.90 

3.76 

0.83 

3.79 

0.87 

Working  Cooperatively 

4.06 

0.86 

4.12 

0.78 

4.13 

0.85 

4.10 

0.81 

Interpersonal  Tolerance 

3.51 

0.97 

3.71 

0.85 

3.76 

0.79 

3.68 

0.86 

Self-Esteem 

3.78 

0.98 

3.75 

0.87 

3.82 

0.85 

3.77 

0.88 
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Table  2.5  Worker  Requirement  Ratings  for  Doing  the  Job  for  the  Three  Options  and  All  ATCSs  (Continued) 


WORKER  REQUIREMENT 
LABEL 

ARTCC 

TERMINAL 

FLIGHT  SERVICE 

ALL  ATCSs 

MEAN 

SD 

MEAN 

SD 

MEAN 

SD 

MEAN 

SD 

Self-Confidence 

4.01 

0.81 

4.00 

0.79 

4.07 

0.77 

4.02 

0.78 

Taking  Charge 

4.43 

0.68 

4.38 

0.68 

4.03 

0.81 

4.30 

0.73 

Self-Awareness 

4.17 

0.82 

4.15 

0.79 

3.94 

0.73 

4.11 

0.78 

Attention  to  Detail 

3.82 

0.86 

3.76 

0.86 

3.84 

0.81 

3.80 

0.85 

Task  Closure/Thoroughness 

3.97 

0.86 

3.91 

0.84 

4.01 

0.82 

3.96 

0.85 

Decisiveness 

4.40 

0.76 

4.45 

0.69 

4.26 

0.75 

4.40 

0.71 

Behavioral  Consistency 

3.65 

0.94 

3.73 

0.85 

3.96 

0.72 

3.76 

0.85 

Flexibility  (S/A) 

4.31 

0.80 

4.21 

0.79 

4.19 

0.72 

4.22 

0.77 

Concentration 

4.21 

0.80 

4.28 

0.78 

4.11 

0.83 

4.23 

0.80 

Composure 

4.60 

0.65 

4.53 

0.69 

4.38 

0.73 

4.50 

0.70 

Tolerance  for  High  Intensity 

4.68 

0.58 

4.66 

0.56 

4.46 

0.68 

4.60 

0.62 

Motivation 

3.45 

1.00 

3.48 

0.91 

3.47 

0.92 

3.47 

0.93 

Commitment  to  the  Job 

3.79 

0.96 

3.90 

0.92 

3.98 

0.88 

3.88 

0.92 

Chunking 

3.44 

0.90 

3.45 

0.94 

3.79 

0.86 

3.53 

0.92 

Mathematical  Reasoning 

3.27 

0.95 

2.99 

1.01 

3.13 

1.01 

3.08 

1.01 

Visuospatial  Reasoning 

3.94 

0.87 

3.90 

0.97 

3..39 

0.93 

3.78 

0.97 

Mechanical  Reasoning 

3.18 

1.04 

3.09 

1.00 

3.27 

0.95 

3.15 

1.00 

Verbal  Reasoning 

3.81 

0.80 

3.74 

0.89 

4.22 

0.78 

3.89 

0.86 

Angles 

3.16 

1.12 

2.74 

1.08 

2.58 

1.18 

2.79 

1.13 

Flexibility  (IP) 

3.83 

0.85 

3.77 

0.84 

3.50 

0.86 

3.72 

0.85 

Dynamic  Visual-Spatial 

3.83 

0.95 

3.99 

0.90 

3.20 

1.12 

3.80 

1.02 

Movement  Detection 

3.54 

1.08 

4.03 

0.89 

3.31 

1.10 

3.80 

1.03 

Perceptual  Speed  &  Accuracy 

4.18 

0.75 

4.20 

0.79 

3.91 

0.90 

4.13 

0.83' 

2-D  Mental  Rotation 

3.05 

1.20 

3.35 

1.04 

2.97 

1.11 

3.20 

1.12 

3-D  Mental  Rotation 

2.90 

1.26 

3.51 

1.09 

2.691 

1.21 

3.24 

1.20 

Realistic  Orientation 

3.07 

1.16 

3.24 

0.97 

3.19 

1.03 

3.19 

1.02 

Internal  Locus  of  Control 

4.00 

0.82 

3.81 

1.10 

3.74 

0.84 

3.83 

0.92 

Recall  From  Interruption 

4.03 

0.83 

3.91 

0.95 

3.85 

0.88 

3.92 

0.90 

Translation  of  Uncertainty 

3.74 

0.96 

3.79 

1.00 

3.37 

1.00 

3.67 

1.01 

Aviation  Science  Background 

3.61 

0.94 

3.56 

0.90 

4.02 

0.83 

3.67 

0.91 

Geography 

3.11 

1.01 

2.90 

0.94 

4.40 

0.77 

3.32 

1.13 
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Table  2.6.  Worker  Requirement  Ratings  for  Learning  the  Job  for  the  Three  Options  and  All  ATCSs 


WORKER  REQUIREMENT 

LABEL 

ARTCC 

TERMINAL 

FLIGHT ! 

SERVICE 

ALL  ATCSs 

MEAN 

SD 

MEAN 

SD 

MEAN 

SD 

MEAN 

SD 

Oral  Communication 

4.26 

0.81 

4.30 

0.87 

4.20 

0.91 

4.25 

0.87 

Written  Communication 

3.29 

0.98 

3.41 

0.85 

3.44 

1.04 

3.39 

0.93 

Reading 

4.03 

0.97 

4.09 

0.85 

4.34 

0.81 

4.15 

0.86 

Active  Listening 

4.45 

0.68 

4.51 

0.66 

4.31 

0.74 

4.43 

0.71 

Translating  Information 

3.82 

0.91 

3.76 

0.91 

4.15 

0.86 

3.87 

0.92 

Interpreting  Information 

3.23 

1.08 

3.59 

0.90 

4.33 

0.81 

3.72 

0.98 

Summarizing  Information 

3.77 

0.87 

3.81 

0.86 

4.24 

0.79 

3.91 

0.86 

Numeric  Ability  (add/sub) 

3.14 

1.07 

2.91 

0.96 

3.00 

0.90 

2.96 

0.96 

Numeric  Ability  (mult/div) 

2.79 

1.21 

2.62 

0.94 

2.61 

0.97 

2.64 

1.00 

Street  Physics 

3.79 

1.01 

3.77 

0.90 

3.38 

0.99 

3.67 

0.97 

Short-Term  Memory 

4.18 

0.85 

4.23 

0.84 

3.98 

0.83 

4.16 

0.85 

Intermediate-Term  Memory 

3.85 

0.96 

3.89 

0.82 

3.87 

0.86 

3.87 

0.87 

Long-Term  Memory 

4.37 

0.76 

4.14 

0.86 

3.89 

0.91 

4.11 

0.88 

Thinking  Ahead 

4.35 

0.79 

4.32 

0.79 

3.54 

1.03 

4.12 

0.92 

Planning 

4.28 

0.75 

4.34 

0.79 

3.89 

0.91 

4.21 

0.84 

Prioritization 

4.46 

0.68 

4.41 

0.80 

3.99 

0.92 

4.30 

0.83 

Execution 

4.33 

0.73 

4.32 

0.79 

3.78 

0.98 

4.18 

0.85 

Problem  Identification 

4.22 

0.76 

4.16 

0.80 

3.71 

0.94 

4.05 

0.85 

Reasoning 

4.21 

0.88 

4.22 

0.76 

3.92 

0.85 

4.13 

0.82 

Creativity 

3.83 

0.95 

3.82 

0.91 

3.27 

1.08 

3.68 

0.99 

Self-Monitoring/Evaluating 

3.91 

1.04 

3.97 

0.90 

3.68 

0.85 

3.89 

0.93 

Rule  Application 

4.27 

0.75 

4.22 

0.86 

4.03 

0.84 

4.19 

0.84 

Sustained  Attention 

3.95 

0.98 

3.95 

0.92 

3.66 

0.99 

3.87 

0.96 

Time  Sharing 

4.17 

0.83 

4.18 

0.84 

3.58 

0.97 

4.01 

0.91 

Situational  Awareness 

4.21 

0.89 

4.21 

0.89 

3.70 

0.98 

4.08 

0.93 

Visualization 

4.09 

0.85 

4.15 

0.91 

3.86 

0.96 

4.06 

0.91 

Projection 

4.13 

0.84 

4.17 

0.87 

3.76 

0.99 

4.05 

0.91 

Scanning 

4.16 

0.87 

4.20 

0.89 

3.96 

0.93 

4.12 

0.90 

Professionalism 

3.71 

1.04 

3.79 

0.90 

3.47 

0.97 

3.69 

0.95 

Working  Cooperatively 

3.83 

1.02 

3.94 

0.86 

3.63 

1.10 

3.82 

0.96 

Interpersonal  Tolerance 

3.71 

0.89 

3.78 

0.90 

3.50 

0.88 

3.68 

0.90 

Self-Esteem 

3.88 

0.95 

3.88 

0.83 

3.73 

0.88 

3.82 

0.87 
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Table  2.6.  Worker  Requirement  Ratings  for  Learning  the  Job  for  the  Three  Options  and  All  ATCSs  (Com.) 


WORKER  REQUIREMENT 

LABEL 

ARTCC 

TERMINAL 

FLIGHT  SERVICE 

ALL  ATCS.S 

MEAN 

SD 

MEAN 

SD 

MEAN 

SD 

MEAN 

SD 

Self-Confidence 

3.91 

0.92 

3.93 

0.84 

3.74 

0.87 

3.87 

0.86 

Taking  Charge 

4.20 

0.92 

4.15 

0.88 

3.64 

0.94 

4.02 

0.92 

Self-Awareness 

3.92 

0.98 

4.03 

0.87 

3.66 

0.88 

3.90 

0.90 

Attention  to  Detail 

3.64 

0.99 

3.64 

0.90 

3.42 

0.91 

3.58 

0.92 

Task  Ciosure/Thoroughness 

3.85 

0.92 

3.88 

0.85 

3.80 

0.86 

3.86 

0.87 

Decisiveness 

4.20 

0.86 

4.23 

0.82 

3.96 

0.79 

4.15 

0.82 

Behavioral  Consistency 

3.71 

0.94 

3.83 

0.77 

3.79 

0.75 

3.78 

0.81 

Flexibility  (S/A) 

4.08 

0.81 

4.10 

0.83 

3.86 

0.82 

4.04 

0.83 

Concentration 

4.00 

0.91 

4.19 

0.78 

3.82 

0.82 

4.06 

0.83 

Composure 

4.35 

0.77 

4.34 

0.78 

4.05 

0.83 

4.27 

0.80 

Tolerance  for  High  Intensity 

4.42 

0.80 

4.43 

0.74 

4.09 

0.83 

4.32 

0.79 

Motivation 

3.69 

0.99 

3.74 

0.89 

3.49 

1.00 

3.67 

0.93 

Commitment  to  the  Job 

3.99 

1.01 

4.03 

0.89 

3.90 

0.93 

3.98 

0.92 

Chunking 

3.43 

0.92 

3.52 

0.93 

3.53 

0.94 

3.50 

0.93 

Mathematical  Reasoning 

3.28 

0.92 

3.09 

1.00 

3.08 

0.99 

3.12 

0.99 

Visuospatial  Reasoning 

3.91 

0.88 

3.86 

0.93 

3.30 

0.90 

3.74 

0.95 

Mechanical  Reasoning 

3.13 

1.04 

3.10 

0.97 

3.18 

0.93 

3.13 

0.98 

Verbal  Reasoning 

3.75 

0.90 

3.73 

0.90 

3.99 

0.74 

3.81 

0.86 

Angles 

3.16 

1.06 

2.82 

1.10 

2.54 

1.13 

2.83 

1.12 

Flexibility  (IP) 

3.71 

0.90 

3.70 

0.85 

3.25 

0.92 

3.59 

0.90 

Dynamic  Visual-Spatial 

3.74 

0.97 

3.85 

0.91 

3.17 

1.04 

3.69 

0.99 

Movement  Detection 

3.50 

1.10 

3.92 

0.95 

3.20 

1.11 

3.70 

1.05 

Perceptual  Speed  &  Accuracy 

4.03 

0.82 

4.09 

0.84 

3.74 

0.93 

3.99 

0.87 

2-D  Mental  Rotation 

3.03 

1.14 

3.35 

1.04 

2.89 

1.10 

3.18 

1.11 

3-D  Mental  Rotation 

2.89 

1.20 

3.43 

1.06 

2.59 

1.21 

3.18 

1.17 

Realistic  Orientation 

3.12 

1.14 

3.23 

0.98 

3.06 

1.05 

3.16 

1.03 

Internal  Locus  of  Control 

3.86 

0.92 

3.76 

0.97 

3.47 

0.90 

3.71 

0.95 

Recall  From  Interruption 

3.93 

0.86 

3.81 

0.93 

3.56 

0.93 

3.77 

0.91 

Translation  of  Uncertainty 

3.55 

0.98 

3.71 

0.96 

3.20 

0.98 

3.55 

0.99 

Aviation  Science  Background 

3.33 

1.09 

3.45 

0.98 

3.71 

1.00 

3.48 

1.02 

Geography 

2.88 

1.05 

2.85 

0.95 

4.01 

1.00 

3.14 

1.12 
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Table  2.7.  Survey  Subactivities  for  All  ATCSs  Ranked  by  the  Mean  Criticality  Index _ 

MEAN  SD  SUBACTIVITY  LABEL 

14.39  1.27  Checking  and  evaluating  separation  or  traffic  movement  to  ensure  separation  is  maintained 

12.96  2.25  Performing  aircraft  conflict  resolution 

12.70  2.37  Establishing  and  maintaining  positive  aircraft  or  vehicle  identification 

12.40  2.40  Establishing  arrival  sequences 

12.34  2.61  Managing  airborne  departures  (e.g.,  observe  takeoff,  issue  go  around) 

1 1.89  2.66  Scanning  to  maintain  awareness  of  surrounding  airspace 

1 1.61  2.61  Issuing  arrival  and  landing  information  or  instructions 

11.51  2.59  Observing  arrival  aircraft 

11.43  2.57  Establishing/maintaining/terminating  radio  communications 

11.37  2.67  Managing  departure  flows 

11.37  2.12  Responding  to  special  conditions,  contingencies,  or  emergencies 

11.34  2.58  Managing  departure  traffic 

11.32  2.46  Managing  personal  workload 

1 1 .3 1  2.58  Issuing  departure  information  or  instructions 

11.26  2.14  Responding  to  contingencies/emergencies 

11.22  2.29  Briefing  relieving  controllers 

11.18  2.43  Assuming  position  responsibility 

11.15  2.68  Recognizing  and  responding  to  deviations  from  ATCS  instructions/clearances 

10.94  2.79  Establishing/re-establishing/terminating  radar  identification 

10.85  2.78  Disseminating  weather  information  to  pilots/other  controllers 

10.58  "  2.68  Conducting  search  and  rescue  procedures 

10.56  2.37  Orienting  lost  aircraft 

10.55  2.81  Prioritizing  sector/position  tasks 

10.54  2.60  Issuing  clearances 

10.46  2.60  Responding  to  pointouts  based  on  current  or  anticipated  traffic  situations 

10.41  2.57  Responding  to  ground  movement  requests 

10.35  2.62  Initiating  pointouts 

10.31  2.64  Responding  to  significant  weather  information 

10.28  2.50  Initiating  search  and  rescue  procedures 

10.04  2.67  Issuing  unsafe  condition  advisories 

10.02  3.05  Performing  procedures  for  non-radar  environment 

9.92  2.88  Responding  to  changes  in  runway  or  taxiway  usage 

9.9 1  2.56  Receiving  transfer  of  control  or  radar  identification 

9.90  3.08  Maintaining  currency  in  weather  data 

(Continued) 
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Table  2.7  Survey  Subactivities  for  AH  ATCSs  Ranked  by  the  Man  Criticality  Index  (Continued) 


MEAN 

SD 

SUBACTIVITY  LABEL 

9.89 

2.38 

Analyzing  initial  requests  for  clearances 

9.83 

3.03 

Managing  aborted  takeoffs 

9.82 

2.55 

Initiating  transfer  of  control  or  radar  identification 

9.81 

2.36 

Responding  to  communications  failures 

9.63 

2.46 

Executing  backup  procedures  for  communications  failures 

9.61 

2.64 

Responding  to  computer  failures 

9.58 

2.68 

Processing  flight  plans  or  flight  plan  amendments 

9.53 

2.55 

Executing  backup  procedures  for  radar  display  failures 

9.50 

2.63 

Transferring  control  responsibilities  for  departure  aircraft 

9.49 

2.46 

Processing  pilot  requests  for  clearance 

9.41 

2.59 

Reviewing  route  of  flight 

9.40 

2.76 

Processing  weather  reports 

9.40 

2.81 

Conducting  direct  weather  observations 

9.37 

2.74 

Responding  to  traffic  management  constraints  or  flow  control  conflicts 

9.32 

2.75 

Managing  automated  handoff  features 

9.30 

2.43 

Responding  to  ARTS/BRITE/FDIO  failures 

9.28 

2.78 

Performing  minimum  safe  altitude  processing 

9.13 

2.40 

Forwarding  flight  data  information 

9.12 

2.83 

Managing  staff  workload  (performing  CIC  duties) 

9.11 

2.37 

Responding  to  flow  constraints 

9.08 

2.54 

Requesting,  processing  and  entering  PIREPS  (pilot  reports) 

9.06 

2.73 

Broadcasting  TIBSfTWEBS/PATWAS/HIWAS 

9.00 

2.69 

Responding  to  position  or  sector  reconfigurations 

8.09 

2.85 

Providing  hazardous  area  reporting  services 

8.96 

2.52 

Executing  backup  procedures  for  ARTS/BRlTE/FDlO  failures 

8.86 

2.53 

Updating  flight  progress  strips 

8.85 

2.50 

Responding  to  movement  area  closures  or  reopenings 

8.85 

2.66 

Responding  to  imposed  airspace  restrictions 

8.82 

3.04 

Receiving  system  status  information  and  automated  weather  information 

8.82 

3.04 

Receiving  system  status  information  and  automated  weather  information 

8.81 

2.53 

Responding  to  imposed  movement  area  restrictions 

8.81 

2.88 

Planning  clearances 

8.78 

2.42 

Responding  to  requests  for  transfer  of  control 

8.75 

2.36 

Operating  airport  or  taxiway  lighting  systems 

(Continued) 
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Table  2.7.  Survey  for  All  ATCSs  Ranked  by  the  Mean  Criticality  Index  (Continued) 


MEAN 

SD 

SUBACTIVITY  LABEL 

8.74 

2.65 

Processing  emergency  locator  transmitter  signals 

8.74 

2.61 

Monitoring  navigational  equipment  and  conununication  equipment 

8.73 

2.66 

Reporting  significant  aeronautical  data 

8.71 

2.74 

Processing  pilot  requests  for  airport  information  (airport  advisories) 

8.68 

2.57 

Performing  movement  area  violation  resolution 

8.68 

2.92 

Processing  aeronautical  data  (e.g.,  NOT  AM’s,  MOA) 

8.51 

2.65 

Executing  backup  procedures  for  loss  of  flight  plan  data  base 

8.50 

3.08 

Activating  flight  plans 

8.49 

2.50 

Executing  backup  procedures  for  NAVAID  failures 

8.45 

2.56 

Responding  to  airport  equipment  failiues 

8.40 

2.65 

Responding  to  pilot  requests  for  route  planning 

8.35 

3.02 

Maintaining  inbound/proposed/suspense  lists 

8.34 

2.63 

Responding  to  special  use  airspace  events 

8.28 

2.63 

Housekeeping  or  sector/position  management 

8.24 

2.33 

Executing  backup  procedures  for  sensor  or  tracking  failures 

8.23 

3.08 

Processing  flight  plan  closures 

8.12 

2.69 

Processing  departure  or  en  route  time  information 

8.02 

2.40 

Responding  to  requests  for  temporary  release  of  airspace 

8.02 

2.35 

Responding  to  sensor  outages 

7.98 

3.10 

Reviewing  proposed/inbound/suspense  lists 

7.94 

2.51 

Responding  to  special  operations  (e.g.,  VIP  movements,  parachutes) 

7.93 

2.56 

Requesting  temporary  release  of  airspace 

7.92 

2.64 

Requesting  temporary  release  of  movement  areas 

7.88 

2.71 

Responding  to  requests  for  temporary  release  of  movement  areas 

7.65 

2.76 

Handling  other  facilities  starting/closing  operations 

7.58 

2.29 

Processing  requests  for  flight  following 

7.49 

2.39 

Suppressing  or  restoring  alerts 

7.45 

2.29 

Editing  data  base  messages 

7.40 

2.70 

Monitoring  non-controlled  objects  (e.g.,  parachutes,  balloons,  animals) 

7.24 

2.65 

Coordinating  with  US  Customs 
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Table  2.8.  Worker  Requirement  Definitions  Used  in  the  Predictor- WR  Linkage  Survey 


Worker 

Requirement 

Definitions 

Prioritization 

The  ability  to  identify  activities  that  are  most  critical  and  require  immediate  attention.  This 
involves  a  constant  evaluation  of  new  information  followed  by  re-prioritization  of  job 
activities. 

Tolerance  for  High 
Intensity 

The  ability  to  perform  effectively  and  think  clearly  during  heavy  work  flow. 

Composure 

The  ability  to  think  clearly  in  stressful  situations. 

Active  Listening 

The  ability  to  hear  and  comprehend  spoken  information.  This  ability  requires  an  individual 
to  recognize  or  pick  out  pertinent  auditory  information. 

Oral 

Communication 

The  ability  to  speak  clearly  and  concisely  to  individuals  so  they  understand  what  is  being 
communicated.  Projecting  a  confident  tone  of  voice  is  an  important  component  of  this 
ability. 

Situational 

Awareness 

Being  cognizant  of  all  information  within  a  four  dimensional  space  (i.e.,  separation 
standards  plus  time).  This  involves  the  ability  to  “understand”  the  airspace  as  an  integrated 
whole  (e.g.,  getting  the  picture). 

Planning 

The  ability  to  determine  the  appropriate  course(s)  of  action  to  take  in  any  given  situation. 

Execution 

The  ability  to  take  timely  action  to  ayoid  problems  and  to  solve,  existing  problems 

Thinking  Ahead 

The  ability  to  anticipate  or  recognize  problems  before  they  occur  and  to  deyelop  plans  to 
ayoid  problems.  This  includes  thinking  about  what  might  happen. 

Taking  Charge 

The  ability  to  take  control  of  a  situation  and  reach  out  and  take  correct  action. 

Reasoning 

The  ability  to  apply  ayailable  information  in  order  to  make  decisions,  draw  conclusions,  or 
identify  alternatiye  solutions. 

Time  Sharing 

The  ability  to  perform  two  or  more  job  actiyities  at  the  same  time. 

Decisiveness 

The  ability  to  make  effectiye  decisions  in  a  timely  manner. 

Short-Term 

Memory 

The  ability  to  remember  pertinent  information  within  a  brief  period  of  time  (less  than  1 
minute).  Examples  of  information  include  call  signs  and  keywords. 

Scanning 

The  ability  to  quickly  and  accurately  search  for  information  on  a  computer  screen,  radar 
scope,  or  computer  print-out. 

Problem  Solving 

The  ability  to  identify  a  potential  problem  or  existing  problem  and  to  identify  the  yariables 
used  in  solying  the  problem. 

Flexibility 

(Stability/ 

Adjustment) 

The  ability  to  adapt  to  changing  situations  or  conditions. 

Long-Term 

Memory 

The  ability  to  remember  pertinent  information  oyer  long  periods  of  time.  Examples  of 
information  include  maps  and  separation  procedures. 

Projection 

The  ability  to  translate  material  into  visual  representation  of  what  will  occur  in  the  future. 

Visualization 

The  ability  to  translate  material  into  a  visual  representation  of  what  is  currently  occurring. 

Concentration 

The  ability  to  focus  on  job  activities  amid  distractions  for  short  periods  of  time. 

Confirmation 

The  ability  to  efficiently  select  a  response  option  consistent  with  the  application  of 
inferred  rules. 

(Continued) 
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Table  2.8.  Worker  Requirement  Definitions  Used  in  the  Predictor- WR  Linkage  Survey  (Continued) 


Worker 

Requirement 

Definitions 

Self-Awareness 

The  ability  to  maintain  an  internal  awareness  of  your  actions  and  attitudes.  This  includes 
knowing  your  limitations. 

Rule  Application 

The  ability  to  efficiently  apply  transformational  rules  inferred  from  the  complete  portions 
of  the  stimulus  array  to  the  incomplete  portion  of  the  array. 

Perceptual  Speed 
&  Accuracy 

Ability  to  perceive  visual  information  quickly  and  accurately  and  to  perform  simple 
processing  tasks  with  (e.g.,  comparisons). 

Sustained  Attention 

The  ability  to  stay  focused  on  a  task(s)  for  long  periods  of  time  (over  60  minutes). 

Self-Monitoring/ 

Evaluation 

The  ability  and  willingness  to  check  your  own  work  performance,  evaluate  the 
effectiveness  of  your  decisions,  and  alter  your  performance  if  necessary. 

Creativity 

The  ability  to  identify  new  or  novel  solutions  to  potential  problems  when  existing  or 
established  solutions  no  longer  apply. 

Working  The  willingness  to  work  with  others  to  achieve  a  common  goal.  This  includes  a 

Cooperatively _ willingness  to  voluntarily  assist  another  controller  if  the  situation  warrants. 


Recall  from  The  ability  to  recall  a  deferred  or  interrupted  action  when  priorities  permit,  and  to  be  able 

Interruption _ to  resume  the  action  appropriately. _ 


Self-Confidence 

- -  r  ..  -J'  - 

A  belief  that  you  are  the  person  for  the  job  and  knowing  that  your  processes  and  decisions 
are  correct. 

Internal  Locus  of 
Control 

Believes  that  individuals  have  influence  over  the  outcome  of  an  event;  takes  responsibility 
for  outcomes. 

Street  Physics 

The  ability  to  understand  the  general  rules  of  science  as  practiced  in  the  ATCS 
environment  (e.g.,  aircraft  size  and  turbulence  effects,  angles  of  intersect,  overtake  speed, 
headings). 

Task  Closure/ 
Thoroughness 

The  ability  to  continue  an  activity  to  completion  through  the  coordination  and  inspection 
of  work. 

Summarizing 

Information 

The  ability  to  summarize  and  consolidate  information  most  relevant  to  the  situation. 

Intermediate-Term 

Memory 

The  ability  to  remember  pertinent  information  over  a  1-10  minute  period. 

Visuospatial  Ability  to  perceive  and  understand  principles  governing  relationships  among  several 

Reasoning _ figures. _ _ _ 


Flexibility  The  ability  to  find  new  meanings  for  stimuli,  to  combine  stimulus  attributes  to  come  up 

(Information  with  new  and  different  solution  protocols,  and  to  employ  flexible  ways  of  relating  new 

Processing) _ information  to  stored  knowledge. _ 


Dynamic  Visual- 
Spatial 

.  . o  •  .  . 

Ability  to  deal  with  dynamic  visual  movement. 

Professionalism 

The  ability  to  establish  respect  and  confidence  in  your  abilities  among  other  controllers. 

Attention  to  Detail 

The  ability  to  recognize  and  attend  to  the  details  of  the  job  that  others  might  overlook. 

Verbal  Reasoning 

The  ability  to  perceive  and  understand  principles  governing  the  use  of  verbal  concepts  and 
symbols. 

Reading 

The  ability  to  read  and  understand  written  information  (e.g.,  ATCS  documents,  manuals). 

Learning 

Changes  in  information-processing  strategies  over  time  or  trials  not  due  to  maturation  or 
aging. 

(Continued) 
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Table  2.8.  Worker  Requirement  Definitions  Used  in  the  Predictor-WR  Linkage  Survey  (Continued) 


Worker 

Requirement 

Definitions 

Commitment  to  the 
Job 

The  desire  to  be  an  ATCS  and  work  hard  to  be  successful. 

Self-Esteem 

Having  a  positive  opinion/image  of  one.self. 

Translation  of 
Uncertainty 

The  ability  to  assign  a  subjective  probability  regarding  the  likelihood  of  an  event 
occurring;  the  ability  to  use  probabilities  to  identify  optimal  courses  of  action. 

Translating 

Information 

The  ability  to  translate  symbols  or  symbolic  abbreviations  into  meaningful  information. 

Behavioral 

Consistency 

The  ability  to  behave  consistently  at  work  (e.g.,  dealing  with  coworkers  in  a  consistent 
manner;  consistently  using  the  correct  phraseology). 

Encoding 

Transformation  or  translation  of  information;  coding;  decoding. 

Movement 

Detection 

Ability  to  detect  physical  movement  of  objects  and  to  judge  their  direction. 

Interpersonal 

Tolerance 

The  ability  to  accommodate  or  deal  with  differences  in  personalities,  criticisms,  and 
interpersonal  conflicts  in  the  work  environment. 

Motivation 

The  desire  to  motivate  one.self  through  challenges  on  the  job  and  to  progress  to  a  higher 
level  of  skill. 

Chunking 

The  ability  to  organize  stimuli  into  meaningful  groups  or  units. 

Interpreting 

Information 

The  ability  to  put  information  into  meaningful  terms.  It  is  the  ability  to  recognize  the 
implications  for  a  statement  or  condition  (e.g.,  cold  front). 

Mathematical 

Reasoning 

Ability  to  perceive  and  understand  principles  governing  the  use  of  quantitative  concepts 
and  symbols. 

Written 

Communication 

The  ability  to  write  legibly  and  accurately  (e.g.,  strip  markings). 

Mechanical 

Reasoning 

Ability  to  perceive  and  understand  the  relationship  of  physical  forces  and  mechanical 
elements  in  a  prescribed  situation. 

Angles 

The  ability  to  apply  the  principles  of  geometry  to  angles  and  computations  involving 
angles.  The  ability  involves  both  the  speed  and  accuracy  of  computation. 

Rule  Inference 

The  ability  to  efficiently  ascertain  the  rules  governing  relations  between  stimulus 
attributes. 

Realistic 

Orientation 

Prefers  dealing  with  activities  that  have  tangible  and  measurable  consequences;  enjoys 
activities  which  require  skill,  is  reinforced  by  accomplishing  realistic  tasks. 

2-D  Mental 

Rotation 

Ability  to  identify  a  two-dimensional  figure  when  seen  at  different  angular  orientations. 

Numeric  Ability 
(add/sub) 

The  ability  to  quickly  and  accurately  perform  basic  math  operations  (addition  and 
subtraction). 

3-D  Mental 

Rotation 

Ability  to  identify  a  three-dimensional  object  when  seen  at  different  angular  orientations 
either  within  the  picture  plane  or  about  the  axis  in  depth. 

Numeric  Ability 
(mult/div) 

The  ability  to  quickly  and  accurately  perform  basic  math  operations  (multiplication  and 
division). 
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Table  2.9.  Number  of  Raters  and  Intra-Class  Correlations  for  Each  Scale 


AT-AT-SAT  Predictor 

Number  of  Raters 

Intra-Class 

Correlation 

Version  1 

Applied  Math 

12 

.95 

Dials 

12 

.91 

Letter  Factory 

12 

.94 

Scanning 

12 

.92 

Sound  Memory 

12 

.94 

Time  Wall 

12 

.92 

Version  2 

Angles 

14 

.89 

Air  Traffic  Scenarios 

14 

.92 

Analogies 

14 

.93 

Memory 

13 

.88 

Planes 

13 

.86 

EQ-Tolerance  for  High  Intensity 

14 

.93 

EQ-Composure 

14 

.95 

EQ-Decisiveness 

14 

.90 

EQ-Execution 

14 

.94 

EQ-Taking  Charge 

14 

.89 

EQ-Concentration 

14 

.93 

EQ-Flexibility 

14 

.89 

EQ-Self-Awareness 

14 

.95 

EQ-Working  Cooperatively 

14 

.96 

EQ-Sustained  Attention 

12 

.94 

EQ-Self-Confidence 

12 

.96 

EQ-Task  Closure/Thoroughness 

12 

.92 

EQ-Interpersonal  Tolerance 

12 

.94 

EQ-Consistency  of  Work  Behavior 

12 

.87 
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Table  2.10,  AT-SAT  Tests  Rated  as  Measuring  Each  SACHA-Generated  Worker  Requirement 


Worker  Requirement  ^ 

Te.st 

Mean  ^ 

SD 

Prioritization 

Air  Traffic  Scenarios 

4.64 

Letter  Factory 

4.25 

.97 

Tolerance  for  High  Intensity 

EQ-Tol.  for  High  Intensity 

4.71 

.47 

Air  Traffic  Scenarios 

4.14 

.86 

Letter  Factory 

3.92 

1.00 

EQ-Composure 

3.79 

.89 

Composure 

EQ-Composure 

4.57 

.85 

EQ-Tol.  for  High  Intensity 

3.43 

1.09 

Air  Traffic  Scenarios 

3.36 

1.01 

Active  Listening 

Sound  Memory 

4.33 

.89 

Oral  Communication 


Situational  Awareness 

Air  Traffic  Scenarios 

4.50 

1.34 

Letter  Factory 

3.33 

1.83 

Planning 

Air  Traffic  Scenarios 

4.14 

.77 

Letter  Factory 

4.00 

.85 

Execution 

Letter  Factory 

4.42 

.67 

Air  Traffic  Scenarios 

4.07 

.73 

EQ-Execution 

3.86 

1.79 

Thinking  Ahead 

Air  Traffic  Scenarios 

4.14 

.66 

Letter  Factory 

3.92 

1.08 

Taking  Charge 

EQ-Taking  Charge 

4.43 

.94 

Reasoning 

Analogies 

3.36 

1.65 

Air  Traffic  Scenarios 

3.09 

1.58 

Time  Sharing 

Time  Wall/Pattem  Recog. 

4.08 

1.56 

Letter  Factory 

3.50 

1.51 

Decisiveness 

EQ-Decisiveness 

4.07 

1.44 

Air  Traffic  Scenarios 

3.57 

1.09 

Letter  Factory 

3.50 

1.09 

EQ-Execution 

3.14 

1.17 

Short-Term  Memory 

Sound  Memory 

4.92 

.29 

Letter  Factory 

3.33 

1.23 

Memory  Test 

3.23 

2.01 

Scanning 

Scanning 

5.00 

0 

Letter  Factory 

4.17 

.72 

Air  Traffic  Scenarios 

3.86 

1.23 

Time  Wall/Pattem  Recog. 

3.83 

1.47 

Dials 

3.42 

1.44 

Problem  Solving 


Flexibility  (Stabil./Adjustment)  EQ-Flexibility  4.29  1 .44 


Long-Term  Memory 


(Continued) 
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Table  2.10.  AT-SAT  Tests  Rated  as  Measuring  Each  SACHA-Generated  Worker  Requirement  (Cont.) 


Worker  Requirement  ^ 

Test 

Mean  ** 

SD 

Projection 

Planes 

3.77 

1.79 

Visualization 

Concentration 

EQ-Concentration 

4.29 

1.44 

Letter  Factory 

3.17 

1.70 

Self-Awareness 

EQ-Self  Awareness 

4.50 

.94 

Rule  Application 
(SME-generated) 

Perceptual  Speed  &  Accuracy 

Time  Wall/Pattem  Recog. 

4.58 

.67 

Scanning 

4.58 

.67 

Dials 

4.33 

.89 

Planes 

3.92 

1.61 

Letter  Factory 

3.75 

.97 

Air  Traffic  Scenarios 

3.21 

1.63 

Sustained  Attention 

EQ-Sustained  Attention 

4.07 

.92 

Self-Monitoring/Evaluation 

EQ-Self  Awareness 

2.92 

2.02 

Creativity 

Working  Cooperatively 

EQ-Working  Cooperatively 

4.64 

.74 

Recall  from  Interruption 

Self  Confidence 

EQ-Self  Confidence 

4.64 

.74 

Internal  Locus  of  Control 

Street  Physics 

Task  Closure/Thoroughness 

EQ-Task  Closure 

4.43 

.94 

Summarizing  Information 

Intermediate-Term  Memory 

Memory  Test 

3.77 

1.92 

Visual-Spatial  Reasoning 

Analogies 

3.57 

1.74 

Flexibility  (Inform.  Processing) 


(Continued) 
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FIGURES  AND  TABLES 


Table  2.10.  AT-SAT  Tests  Rated  as  Measuring  Each  SACHA-Generated  Worker  Requirement  (Cont.) 


Worker  Requirement  ^ 

Test 

Mean  ^ 

SD 

Dynamic  Visual-Spatial 

Air  Traffic  Scenarios 

4.07 

1.44 

Planes 

3.54 

1.66 

Time  Wall/Pattem  Recog. 

3.50 

1.45 

Letter  Factory 

3.50 

1.51 

Scanning 

3.08 

1.73 

Professionalism 

Attention  to  Detail 

Letter  Factory 

2.50 

1.57 

Verbal  Reasoning 

Reading 

Commitment  to  the  Job 

Self-Esteem 

EQ-Self  Confidence 

3.21 

1.63 

Translation  of  Uncertainty 

Translating  Information 

Behavioral  Consistency 

EQ-Consist.  Of  Work  Behavior 

3.43 

1.87 

Movement  Detection 

Air  Traffic  Scenarios 

4.29 

.91 

Time  Wall/Pattem  Recog. 

1.78 

Interpersonal  Tolerance 

EQ-Interpersonal  Tolerance 

4.29 

1.44 

EQ-Working  Cooperatively 

3.00 

1.76 

Motivation 


Chunking 


Interpreting  Information 


Mathematical  Reasoning  Applied  Math 


4.83  .39 


Written  Communication 


Mechanical  Reasoning 


Angles  Angles  4.64  .84 

Realistic  Orientation 


(Continued) 


84 


FIGURES  AND  TABLES 


Table  2.10.  AT-SAT  Tests  Rated  as  Measuring  Each  SACHA-Generated  Worker  Requirement  (Cont.) 


Worker  Requirement  ^ 

Test 

Mean  ^ 

SD 

2-D  Mental  Rotation 

Numeric  Ability  (add/sub) 

Applied  Math 

4.58 

.90 

3-D  Mental  Rotation 

Numeric  Ability  (mult./div.) 

Applied  Math 

4.25 

1.48 

Confirmation  ^ 

Analogies 

3.64 

1.69 

Letter  Factory 

3.17 

1.53 

Encoding  ^ 

Memory  Test 

3.00 

1.47 

Rule  Inference  ^ 

Analogies 

4.79 

.43 

Rule  Application  ^ 

Analogies 

4.29 

1.38 

Learning  ^ 

a 

All  but  five  of  the  Worker  Requirements  are  rank  ordered  by  mean  ratings  (of  ARTCC  controllers  from 
SACHA  job  analysis)  regarding  importance  for  doing  the  job.  SACHA  did  not  collect  data  on  the  last  five 
worker  requirements  (Confirmation,  Encoding,  Rule  Inference,  Rule  Application  (SACHA-generated),  and 
Learning)  because  it  was  felt  that  the  complex  cognitive  nature  of  these  WR  definitions  would  not  be  fully 
graspable  by  controllers  completing  the  SACHA  job  analysis  surveys. 

A  criterion  cutoff  mean  >  3  was  established  to  indicate  whether  a  test  was  able  to  measure  a  particular  WR. 
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Table  2.11.  Indicators  of  the  Success  of  AT-SAT  Measures  in  Measuring  Multiple  Worker  Requirements 


AT-SAT  Predictor 

Number  of 
WRs  With 
Ratings  >  3 

Number  of  WRs 
With  Ratings  >  3 
Measured  by  1  or 
Fewer  Tests 

Number  of  WRs 
With  Ratings  > 

3  Not  Measured 
by  Other  Tests 

Mean  of 
Linkage 
Rating  x 
SACHA 
Rating 

Sum  of 
Linkage 
Rating  x 
SACHA 
Rating 

Applied  Math 

3 

3 

3 

13.7 

41.2 

Dials 

2 

0 

0 

16.5 

32.9 

Letter  Factory 

14 

7 

0 

16.2 

226.4 

Scanning 

3 

0 

0 

17.5 

52.5 

Sound 

2 

1 

1 

20.7 

41.3 

Time  Wall 

5 

2 

0 

15.6 

77.9 

Angles 

1 

1 

1 

14.7 

14.7 

AT  Scenarios 

12 

5 

0 

17.5 

209.4 

Analogies 

5 

5 

4 

15.6 

78.2 

Memory 

3 

2 

2 

13.4 

40.1 

Planes 

3 

1 

1 

15.3 

46.0 

EQ-Tol.  for  High  Int. 

2 

0 

0 

18.4 

36.8 

EQ-Composure 

2 

0 

0 

19.4 

38.8 

EQ-Decisiveness 

1 

0 

0 

17.9 

17.9 

EQ-Execution 

2 

0 

0 

15.6 

31.2 

EQ-Taking  Charge 

1 

1 

1 

19.6 

19.6 

EQ-Concentration 

1 

1 

0 

18.1 

18.1 

EQ-Flexibility 

1 

1 

1 

18.5 

18.5 

EQ-Self-A  wareness 

1 

1 

1 

18.9 

18.9 

EQ-Working  Coop. 

1 

1 

1 

18.8 

18.8 

EQ-Sustained  Attent. 

1 

1 

1 

16.8 

16.8 

EQ-Self-Confidence 

2 

2 

2 

15.4 

30.7 

EQ-Task  Closure 

1 

1 

1 

17.6 

17.6 

EQ-Interperson.  Tol. 

1 

1 

1 

15.1 

15.1 

EQ-Consis.  Of  Work 
Behavior 

1 

1 

1 

12.5 

12.5 
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Table  3.1.1.  Regression  Coefficients  for  PTS  Pre-Training  Screen 


Variable 

Beta 

St.  Err. 
of  Beta 

B 

St.  Err. 
ofB 

T(351) 

p-Level 

Static  Vector  %  Correct 

.1337 

.0507 

.2481 

.0341 

2.636 

.0088 

Static  Vector  Correct 

-.1494 

.0601 

-.0054 

.0022 

-2.485 

.0134 

Reaction  Time 

Continuous  Memory 

-.0501 

.0551 

-.0014 

.0015 

-.909 

.3639 

Correct  Reaction  Time 

Time  Wall  Absolute 

-.0546 

.0527 

-.0014 

.0016 

-1.035 

.3012 

Time  Error 

Pattern  Recognition 

-.0166 

.0506 

-.0025 

.0075 

-.239 

.7427 

Correct  Reaction  Time 

ATST  Safety 

-.3068 

.0536 

-.6992 

.1221 

-5.725 

.0000 

Table  3.1.2  Regression  Table  for  Pre-Training  Screen 


Dependent  Variable 

Academy  Screen  Comprehensive  Test  Score 

Multiple  R 

.4906 

Multiple  R-Square 

.2404 

Adjusted  R-Square 

.2277 

Minimum  Pairwise  N 

358 

F  (6,351)=  18.54 

p  <  .0000 

Standard  Error  of  Estimate 

10.59 

Intercept 

72.07 

Standard  Error 

10.50 1(351)  =  6.860  p<.0000 
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Table  3.1.4  Proposed  New  Measures  for  the  ^  WR  Constructs 


CONSTRUCT  WORKER  REQUIREMENT 

CATEGORIES  CONSTRUCTS 


Applied  Reasoning 


Chunking 

Translating  Information 
Interpreting  Information 
Summarizing  Information 
Reasoning  (Thinking) 
Visuospatial  Reasoning 
Verbal  Reasoning 
Mechanical  Reasoning 
Mathematical  Reasoning 
Creativity 
Rule  Application 
Translation  of  Uncertainty 
into  Probability 
Street  Physics 


PRELIMINARY 

TESTS 


PROPOSED 

MEASURES 


Grouping  Info.  Test 


Reasoning  Battery 
Syllogisms 
Analogies 
Classifications 


Rule  Application  Test 


Revise  Available  Street 
Physics  Test 


Computational 


Numeric  Ability  (Addit./Sub.)  ~| _ |~  Headings  Test 

l_  Time  Test 

Numeric  Ability  (Mult./Div.) 

Angles  n - r~  Angles 


Spatial 


Visualization 

Projection 

Dynamic  Visual-Spatial 


_ - _  Projection  Test  — [[^Revise  Paper  Fold.  Test 

_ ~  Ships/Planes  Test  “ 

L  Direct.  &  Distance 
_ - _  Ships/Planes  Test 


Perceptual 


Scaiming 

Movement  Detection 
Perceptual  Speed  &  Accuracy 


Dial  Reading  Test 


Memory 


Short-Term  (working  memory)  ~| _ [~  2  Short-Term  Tests 

1_  Direct.  &  Distance 

Intermediate-Term  Memory  -  Short-Term  Test 

L  (Repeated) 

Long-Term  Memory  ~] -  Map  Memory 


Communication 


Oral  Communication 
Active  Listening 
Written  Communication 
Reading 


Consider  Revision  of 
Ship  Test 

Reading  Test 


We  renamed  the  Ships  test  to  Planes  for  face  validity  puiposes. 


Table  3.1.5.  Proposed  New  Measures  for  the  Processing  Operations  WR  Constructs _ 

CONSTRUCT  WORKER  REQUIREMENT  PRELIMINARY  PROPOSED 

CATEGORIES _ CONSTRUCTS _ TESTS _ MEASURES 

Perceptual  Scanning  A  Complex  Stimuli  Test  “ 

Movement  Detention 

Perceptual  Speed  and  Accuracy  - \Z  Dial  Reading  Test 


Information  Encoding 

Processing  Rule  Inference 

Rule  Application 
Confirmation 
Learning 
Flexibility 


Reasoning  Battery 
Info.  Process 
Measures 


U - STIX  Test 


Metacognitive  Thinking  Ahead 

Problem  Identification 
Planning 
Prioritization 
Self-Monitoring 
Situational  Awareness 


NONE 


Analyze  ATST  for 
Measurement  Possibilities 


Attention 


Sustained  Attention 
Recall  From  Interruption 

Timesharing 


_ - Ships/Planes  Test^ 


-C  Add  scale  to  PEAQ  ‘ 
"C  Develop  Test 


Parallel  Ships  Test  for 
Multitasking  aspect  of 
LJimesharing 


”  Proposed  new  measure  completed. 

The  SACHA  team  completed  an  investigation  of  the  ATST  for  measurement  possibilities.  We  are  currently 
designing  a  metacognitive  test  based  on  this  analysis. 

We  renamed  the  Ships  test  to  Planes  for  face  validity  purposes. 
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Table  3.1.6.  Temperament/Interpersonal  Model 


Big  Five 


T  emperament/Interpersonal 
Construct  Categories  (Hough’s 


Temperament/Interpersonal 

Constructs 


Affiliation 

[none] 

Extraversion 

Potency 

Professionalism  “  (I) 

Taking  Charge  *’  (W) 

Self-Esteem  (S) 

Conscientiousness 

Achievement 

— > 

Self-Esteem  ’’  (S) 

Self-Confidence  ^  (S) 

Execution (W) 

Task  Closure/Thoroughness  ’’  (W) 
Motivation  (W) 

Commitment  to  the  Job  (W) 

Dependability 

- 

Task  Closure/Thoroughness (W) 
Behavioral  Consistency  (W) 

Working  Cooperatively (I) 

Neuroticism 

Adjustment 

- 

Self-Esteem  ’’  (S) 

Flexibility  (A) 

Composure  (A) 

Tolerance  for  High  Intensity 

Work  Situations  (A) 

Agreeableness 

Agreeableness 

Working  Cooperatively (I) 
Interpersonal  Tolerance  (I) 

Openness 

Intellectance 

- 

[none] 

Rugged  Individualism 

Decisiveness  (W) 

Realistic  Orientation  (W) 

Locus  of  Control 

Taking  Charge (W) 

Execution  (W) 

Internal  Locus  of  Control  (S) 

Focus 

— > 

Self  Awareness  (S) 

Concentration  (W) 

Attention  to  Detail  (W) 

“  The  letters  included  in  the  parentheses  indicate  SACHA’s  initial  worker  requirement  categorization  (W  = 
Work  and  Effort,  I  =  Interpersonal,  S  =  Self-Efficacy,  A  =  Stability/Adjustment). 

Included  in  more  than  one  higher-order  construct. 

‘  Not  part  of  Hough’s  taxonomy. 
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EXHIBIT  3.2.1 

PILOT  TEST  ADMINISTRATION 
TEST  BLOCK  SEQUENCING 

GROUPS 

SESSION  1 

SESSION  2 

SESSION  3 

SESSION  4 

SESSION  5 

Group  1 

Block 

Block 

Block 

Block 

Ending  Block  1 

Computers 

1  - 10 

A 

B 

C 

D 

LET 

Group  2 

Block 

Block 

Block 

Block 

Ending 

Computers 

11-20 

B 

E 

A 

C 

Block  2 

ATS 

Group  3 

Block 

Block 

Block 

Block 

Ending 

Computers 

21-30 

C 

A 

D 

E 

Block  3 

SVCM,  Word 
Memory 

Group  4 

Block 

Block 

Block 

Block 

Ending 

Computers 

31  -40 

D 

C 

E 

B 

Block  4 

Word  Memory, 
TWPR 

Group  5 

Block 

Computers 

41-50 

E 

D 

B 

A 

Block  1 

LET 

EXHIBIT  3.2.2 

AIR  TRAFFIC  SCENARIOS  TEST 

EXAMPLE  OF  THE  PLANE  DESCRIPTORS 

F 

3 

B 

□ 

□ 

□ 

The  first  letter 

The  number  indicates 

The  second  letter 

indicates  the  speed. 

The  level.  There  are 

indicates  the 

E  =  East 

four  levels.  “1”  is 

destination.  “A”, 

M  =  Medium 

the  lowest  (closest  to  the 

“B”,  “C”,  and  “D” 

S  =  Slow 

ground)  while  “4”  is  the 

are  exits  out  of 

Highest  (furthest  from 

your  airspace.  “E” 

The  ground). 

and  “E”  are  airports. 
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EXHIBIT  3.23 

SUMMARY  OF  THE  PROPOSED  REVISIONS 
TO  THE  AT-SAT  PILOT  TEST 


TEST  BLOCK 

TESTS 

SUGGESTED  REVISIONS 

BLOCK  A 

Air  Traffic  Scenarios 

□  Clarify  general  test  instructions 

□  Clarify  the  meaning  of  and  action  to  take  on  the  new  planes 

□  Modify  the  countdown 

□  Further  clarify  the  purpose  and  position  of  the  landing  heading 
indicator 

BLOCK  B 

Sound  Test 

□  Clarify  test  instructions 

□  Clarify  the  keys  to  use  for  the  sound  level  adjustment 

□  Instruct  participants  to  remove  the  headphones  after  the  test 

Letter  Factory  Test 

□  Clarify  test  instructions 

□  Clarify  the  demonstration 

□  Clarify  the  instructions  for  the  mouse  exercise 

□  Put  a  time  limit  on  the  mouse  exercise 

BLOCK  C 

Dials  Test 

□  No  changes  needed 

Static  Vector/ 
Continuous  Memory 
Test 

□  Clarify  test  instructions 

□  Explain  and  familiarize  the  participants  with  the  keys  needed  to 
answer  the  scenarios 

Experiences 

Questionnaire 

□  No  changes  needed 

BLOCK  D 

Time  Wall/ 

Pattern  Recognition 
Test 

□  Relocate  the  broken  wall 

□  Explain  and  familiarize  the  participants  with  the  keys  needed  to 
answer  the  scenarios 

Analogy  Test 

□  Reduce  the  required  level  of  vocabulary  and  reasoning  skills 

Classification  Test 

□  Reduce  the  required  level  of  vocabulary  and  reasoning  skills 

BLOCK  E 

Word  Memory  Test 

□  Remove  the  scoring  for  the  test  items 

□  Remove  the  extraneous  window  prompts 

Scan  Test 

□  Clarify  the  test  instructions 

Planes  Test 

□  Reduce  the  length  of  the  practice  session 

□  Modify  the  keystrokes  for  “True”  and  “False” 

□  Remove  the  scoring  for  each  test  item 

Graph  continued  on  next  page. 
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EXHIBIT  3.2.3  CONTINUED 

SUMMARY  OF  THE  PROPOSED  REVISIONS 

TO  THE  AT-SAT  PILOT  TEST 

BLOCK  E 

Angles  Test 

L:  None 

Applied  Mathematics 
Test 

[1  Clarify  the  test  instructions 

J  Reduce  the  level  of  required  mathematics  skills 

ENDING 
BLOCK 
(Groups  1, 5) 

Letter  Factory  Test 

□  Clarify  test  instructions 

L  Clarify  the  demonstration 

Hi  Clarify  the  instructions  for  the  mouse  exercise 

□  Put  a  time  limit  on  the  mouse  exercise 

ENDING 
BLOCK 
(Group  2) 

Air  Traffic  Scenarios 
Test 

□  Clarify  general  test  instructions 

□  Clarify  the  meaning  of  and  action  to  take  on  the  new  planes 

D  Modify  the  countdown 

□  Further  clarify  the  purpose  and  position  of  the  landing  heading 
indicator 

ENDING 
BLOCK 
(Group  3) 

Static  Vector/ 
Continuous  Memory 
Test  &  Word 

Memory 

l'  Clarify  test  instmctions  (SV) 

r.  Explain  and  familiarize  the  participants  with  the  keys  needed  to 
answer  the  scenarios  (SV) 

7  Remove  the  scoring  for  the  test  items  (WM) 

77  Remove  the  extraneous  window  prompts  (WM) 

ENDING 
BLOCK 
(Group  4) 

Word  Memory  & 

Time  Wall/  Pattern 
Recognition  Test 

L7  Remove  the  scoring  for  the  test  items  (WM) 

77i  Remove  the  extraneous  window  prompts  (WM) 
u  Relocate  the  broken  wall  (TW) 

77  Explain  and  familiarize  the  participants  with  the  keys  needed  to 
answer  the  scenarios  (TW) 
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Table  3.3.1.  Item  Analyses  and  Scale  Reliabilities:  Non-Semantic  Word  Scale  on 
the  Analogy  Test  (N=439) _ 


Item 

Number 

Corrected 

Item-Total 

Correlation 

Rerun  of  Corrected 
Item-Total 
Correlation 

Information-Processing 

Corrected  Item-Total 
Correlation® 

Total  Total  Item 

Screen  Response  Time 

Views 

1 

.08 

— 

— 

— 

2 

.16 

.17 

.36 

.46 

4 

-.01 

— 

— 

— 

6 

.19 

.18 

.45 

.49 

8 

.19 

.21 

.36 

.44 

11 

.17 

.18 

.44 

.56 

12 

.14 

.15 

.43 

.53 

14 

.00 

— 

— 

— 

16 

.09 

.16 

.30 

.45 

18 

.03 

— 

— 

— 

20 

.22 

.23 

.37 

.42 

22 

.09 

— 

— 

— 

24 

.27 

.26 

.34 

.46 

28 

-.01 

— 

— 

— 

30 

.15 

.12 

.38 

.49 

Alpha  =  .39 

Alpha  =  .44 

Alpha  =  .84 

15  Items 

9  Items 

9  Items 

“Standardized  score 
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Table  3.3.2.  Item  Analyses  and  Scale  Reliabilities:  Semantic  Word  Scale  on  the  Analogy  Test  (N=439) 


Item 

Corrected 

Rerun  of  Corrected 

Information-Processing 

Number 

Item-Total 

Item-Total 

Corrected  Item  Total 

Correlation 

Correlation 

Correlation® 

Total  Screen  Total  Item 

Views  Response  Time 

3 

.03 

— 

_ 

5 

.24 

.24 

.31 

.38 

7 

.09 

— 

— 

_ 

9 

.15 

.17 

.32 

.41 

10 

.13 

.13 

.30 

.39 

13 

.20 

.19 

.29 

.40 

15 

.14 

.14 

.40 

.54 

17 

.33 

.34 

.30 

.42 

19 

.12 

.11 

.33 

.45 

21 

.13 

.13 

.42 

.52 

23 

.20 

.20 

.35 

.40 

25 

.23 

.23 

.38 

.49 

26 

.28 

.31 

.34 

.45 

27 

.02 

— 

_ 

29 

.33 

.33 

.39 

.42 

Alpha  =  .50 

Alpha  =  .52 

Alpha  = .84 

15  Items 

12  Items 

24  Items 

“Standardized  score 
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Table  3.3.3.  Item  Analyses  and  Scale  Reliabilities;  Semantic  Visual  Scale  on  the  Analogy  Test  (N=439) 


Item 

Number 

Corrected 

Item-Total 

Correlation 

Rerun  of  Corrected 
Item-Total 
Correlation 

Information-Processing 
Corrected  Item-Total 
Correlation® 

Total  Screen  Total  Item 

Views  Response  Time 

31 

.20 

.22 

.33 

.40 

34 

.19 

.23 

.41 

.43 

35 

.16 

.13 

.40 

.42 

37 

.14 

.17 

.40 

.28 

39 

.24 

.26 

.39 

.30 

41 

.16 

.15 

.47 

.42 

44 

.04 

— 

— 

— 

47 

.06 

— 

— 

— 

50 

.26 

.28 

.35 

.36 

51 

.09 

— 

— 

— 

Alpha  =  .40 

Alpha  =  .44 

Alpha  =  .77 

10  Items 

7  Items 

14  Items 

^Standardized  score 
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Table  3.3.4.  Item  Analyses  and  Scale  Reliabilities:  Non-Semantic  Visual  Scale  on  tbe 
Analogy  Test  (N=439) 


Corrected 

Rerun  of  Corrected 

Information-Processing 

Item 

Item-Total 

Item-Total 

Corrected  Item-Total 

Number 

Correlation 

Correlation 

Correlation® 

Total 

Total  Item 

Screen 

Response  Time 

Views 

32 

.25 

.25 

.56 

.56 

33 

.26 

.26 

.61 

.57 

36 

.26 

.26 

.66 

.51 

38 

.26 

.26 

.52 

.48 

40 

.40 

.40 

.43 

.49 

42 

.41 

.41 

.57 

.52 

43 

.37 

.37 

.49 

.54 

45 

.22 

.22 

.69 

.69 

46 

.35 

.35 

.56 

.60 

48 

.30 

.30 

.62 

.61 

49 

.18 

.18 

.62 

.66 

52 

.37 

.37 

.52 

.52 

53 

.18 

.18 

.52 

.57 

Alpha  =  .67 

Alpha  =  .67 

Alpha  = .93 

1 3  Items 

1 3  Items 

26  Items 

“Standardized  score 


Table  3.3.5.  Distribution  of  Test  Completion  Times  for  the  Analogy  Test 


Percentile 

Test  Completion  Time  (Minutes) 

Minimum 

6.1 

5'” 

10.1 

25“’ 

14.9 

50'“ 

18.8 

75“’ 

24.3 

95'“ 

32.1 

Maximum 

35.6 

Mean 

19.8 

SD 

6.59 
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Table  3.3.6.  Estimates  of  Test  Length  to  Increase  Reliability  of  the  Analogy  Test 


Current  Test 

No.  Items  Needed  for 
Alpha  = 

Test  Part 

No.  Items 

Alpha 

.90 

.80 

Non-Semantic  Visual 

13 

.67 

59 

26 

Non-Semantic  Words 

9 

.44 

102 

46 

Semantic  Visual 

6 

.42 

76 

34 

Semantic  Words 

12 

.52 

98 

44 

Total  No.  Items 

40 

335 

150 

Estimated  Test  Time 

190  minutes 

85  minutes 

Table  3.3.7.  Item  Analyses  and  Scale  Reliabilities:  Non-Semantic 
Word  Scale  on  the  Classification  Test  (N=427) 


Item 

Number 

Corrected  Item- 
Total  Correlation* 

1 

.03 

4 

.06 

5 

.12 

6 

.07 

8 

.03 

9 

.03 

13 

.06 

14 

.07 

16 

.05 

18 

.00 

21 

.08 

22 

.05 

Alpha  =  .10 

12  Items 

““Standardized  score 
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Table  3.3.8.  Item  Analyses  and  Scale  Reliabilities:  Semantic  Word  Scale  on  the  Classification  Test  (N-427) 


Item 

Corrected 

Rerun  of  Corrected 

Information-Processing 

Number 

Item-Total 

Item-Total 

Corrected  Item-Total 

Correlation 

Correlation 

Correlation® 

Total  Screen 

Total  Item 

Views 

Response  Time 

2 

.04 

— 

— 

_ 

3 

.21 

.23 

.48 

.36 

7 

.20 

.21 

.40 

.45 

10 

.22 

.19 

.42 

.40 

11 

.10 

.13 

.32 

.51 

12 

.16 

.16 

.41 

.45 

15 

.01 

— 

- _ 

17 

.22 

.21 

.41 

.48 

19 

.26 

.28 

.30 

.51 

20 

.28 

.28 

.20 

.32 

23 

.22 

.23 

.44 

.43 

Alpha  =  .46 

Alpha  =  .49 

Alpha  =  .82 

1 1  Items 

9  Items 

1 8  Items 

“Standardized  score 


Table  3.3.9.  Item  Analyses  and  Scale  Reliabilities:  Non-Semantic  Visual  Scale  on  the 
Classification  Test  (N=427) 


Item 

Number 

Corrected 

Item-Total 

Correlation 

Rerun  of  Corrected 
Item-Total 
Correlation 

Information-Processing 

Corrected  Item-Total 
Correlation® 

Total  Total  Item 

Screen  Response  Time 

Views 

24 

.10 

.10 

.34 

.40 

27 

.09 

.10 

.48 

.60 

28 

.04 

— 

_ 

_ _ 

31 

.23 

.24 

.48 

.51 

33 

.03 

— 

_ 

— 

36 

.02 

— 

_ 

37 

.10 

.13 

.68 

.60 

38 

.17 

.18 

.46 

.52 

39 

.15 

.14 

.29 

.24 

42 

.10 

.07 

.49 

.51 

43 

.15 

.14 

.51 

.53 

45 

.15 

.17 

.41 

.46 

46 

.25 

.25 

.52 

.41 

Alpha  =  .36 

Alpha  =  .40 

Alpha  =  .87 

13  Items 

10  Items 

20  Items 

“Standardized  score 
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Table  3.3.10.  Item  Analyses  and  Scale  Reliabilities:  Semantic  Visual  Scale  on  the  Classification  Test  (N=427) 


Item 

Number 

Corrected 

Item-Total 

Correlation 

Rerun  of  Corrected 
Item-Total 
Correlation 

Information-Processing 

Corrected  Item-Total 
Correlation® 

Total  Total  Item 

Screen  Response  Time 

Views 

25 

.14 

.15 

.46 

.35 

26 

.06 

— 

— 

29 

.lO'’ 

— 

— 

— 

30 

.05 

— 

— 

— 

32 

.06 

— 

— 

_ 

34 

-.01 

— 

— 

— 

35 

-.08 

— 

— 

— 

40 

.05 

— 

— 

— 

41 

.11 

.09 

.51 

.44 

44 

.13 

.12 

.37 

.41 

Alpha  =  .18 

Alpha  =  .23 

Alpha  =  .69 

10  Items 

3  Items 

6  Items 

^Standardized  score 

'’The  alpha  was  higher  excluding  this  item. 


Table  3.3.11.  Distribution  of  Test  Completion  Times  for  the 
Classification  Test  (N=427) 


Percentile 

Test  Completion  Time 
(Minutes) 

Minimum 

2.9 

5“’ 

6.3 

25* 

9.6 

50* 

12.0 

75* 

15.8 

95* 

21.7 

Maximum 

25.4 

Mean 

12.86 

SD 

4.69 
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Table  3.3.12.  Estimates  of  Test  Length  to  Increase  Reliability  of  the  Classification  Test 


Test  Part 

Current  Test 

No.  Items  Needed  for 
Alpha  = 

No.  Items 

Alpha 

.90 

.80 

Non-Semantic  Visual 

10 

.40 

138 

61 

Semantic  Visual 

3 

.23 

92 

41 

Semantic  Words 

9 

.50 

83 

37 

Total  No.  Items 

22 

313 

139 

Estimated  Test  Time 

138  minutes 

63  minutes 

Table  3.3.13.  Planning/Thinking  Ahead:  Distribution  of  Total  Number 
Correct  on  the  Letter  Factory  Test  (Form  A)  (N  =  441) 

Percentile 

Number  of  Attempts 

Minimum 

0 

5'” 

35 

25“’ 

56 

50'“ 

67 

75'“ 

74 

95'“ 

80 

Maximum 

85 

Mean 

63.39 

SD 

14.64 

Table  3.3.14.  Distribution  of  Number  of  Inappropriate  Attempts  to  Place 
a  Box  in  the  Loading  Area  on  the  Letter  Factory  Test  (Form  A)  (N  =  441) 

Percentile 

Number  of  Attempts 

Minimum 

0 

5'“ 

2 

25'“ 

7 

50'“ 

13 

75'“ 

25 

95'“ 

99 

Maximum 

448 

Mean 

26.78 

SD 

46.27 
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Items  (composite  scores 


Table  3.3.17.  Situational  Awareness  (SA)  -  Reliability  Analysis;  Three  Scales  on  the  Letter 
Factory  Test  (Form  A)  (N  =  405) 


Sequence  N( 
(Item  No.) 

Before  Item 
Deletion 

After  Item  Deletion 

3.  Corrected 

Item-Total 
Correlation 

Corrected  Alpha  if  Item 

Item-Total  Deleted 

Correlation 

SA  Level  1 

2(1) 

.14 

.14 

.41 

2(2) 

.25 

.24 

.36 

4(1) 

-.09 

— 

4(2) 

.17 

.21 

.37 

6(1) 

.15 

.18 

.38 

6(2) 

.07 

— 

— 

6(3) 

.18 

.20 

.38 

6(4) 

.13 

.19 

.38 

8(1) 

.02 

— 

— 

8(2) 

-.12 

— 

— 

8(3) 

.06 

— 

— 

8(4) 

.10 

.11 

.42 

11(1) 

.17 

.17 

.39 

11(2) 

.09 

— 

— 

Alpha  =  .31 

14  items 

Alpha  =  .42  8  items 

SA  Level  2 

1(1) 

.32 

.33 

.58 

1(2) 

.30 

.30 

.58 

1(3) 

.38 

.41 

.55 

1(4) 

.20 

.23 

.60 

5(1) 

.31 

.35 

.57 

5(2) 

.26 

.26 

.59 

5(3) 

.01 

___ 

— 

5(4) 

-.03 

— 

— 

9(1) 

.27 

.28 

.59 

9(2) 

.31 

.34 

.57 

9(3) 

.15 

.15 

.61 

9(4) 

.04 

— 

— 

10(1) 

-.01 

— 

— 

10(2) 

.20 

.18 

.61 

10(3) 

.11 

.12 

.62 

10(4) 

.09 

— 

— 

Alpha  =  .53 

16  items 

Alpha  =  .61  11  items 

(Continued) 
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Table  3.3.17.  Situational  Awareness  (SA)  -  Reliability  Analysis:  Three  Scales  on  the  Letter 
Factory  Test  (Form  A)  (N  =  405)  (Continued) _ 


Sequence  No. 
(Item  No.) 

Before  Item 
Deletion 

After  Item  Deletion 

Corrected 

Item-Total 

Correlation 

Corrected 

Item-Total 

Correlation 

Alpha  if  Item 
Deleted 

SA  Level  3  2(3) 

.21 

.11 

.49 

2(4) 

.05 

— 

— 

3(1) 

.20 

.24 

.42 

3(2) 

.21 

.30 

.42 

3(3) 

.24 

.33 

.38 

3(4) 

.18 

.24 

.42 

4(3) 

.04 

— 

— 

4(4) 

.01 

— 

— 

7(1) 

.07 

— 

— 

7(2) 

.00 

— 

— 

7(3) 

.07 

— 

— 

7(4) 

.04 

— 

— 

11(3) 

.11 

.13 

.47 

11(4) 

.17 

.26 

.42 

Alpha  =  .32  14  items 

Alpha  =  .47  7  items 
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Table  3.3.18.  Situational  Awareness  (SA)  -  Reliability  Analysis:  One  Scale  on  the  Letter  Factory  Test 
(Form  A)  (N  =  405) 


Sequence  No. 
(Item  No.) 

Item  Difficulty 

Corrected  Item- 
Total 

Correlation 

Alpha  if  Item 
Deleted 

SA  Level  1 

2(1) 

.47 

.08 

.69 

2(2) 

.33 

.29 

.67 

4(2) 

.45 

.23 

.67 

6(1) 

.40 

.17 

.68 

6(3) 

.47 

.27 

.67 

6(4) 

.49 

.15 

.68 

8(4) 

.38 

.13 

.68 

11(1) 

.76 

.20 

.68 

SA  Level  2 

1(1) 

.67 

.27 

.67 

1(2) 

.62 

.28 

.67 

1(3) 

.45 

.38 

.66 

1(4) 

.46 

.17 

.68 

5(1) 

.48 

.34 

.66 

5(2) 

.45 

.28 

.67 

9(1) 

.45 

.29 

.67 

9(2) 

.35 

.39 

.66 

9(3) 

.17 

.16 

.68 

10(2) 

.61 

.24 

.67 

10(3) 

.18 

.11 

.68 

SA  Level  3 

2(3) 

.45 

.23 

.67 

3(1) 

.36 

.22 

.67 

3(2) 

.09 

.25 

.67 

3(3) 

.62 

.34 

.66 

3(4) 

.37 

.18 

.68 

11(3) 

.14 

.11 

.68 

11(4) 

.21 

.10 

.68 

Note.  Scale  Statistics:  Mean  =  10.87  ; 

SD  =  4.05;  Alpha  = 

.68  (26  items) 

Table  3.3.19.  Planning/Thinking  Ahead:  Distribution  of  Total  Number 
Correct  on  the  Letter  Factory  Test  (Form  B)  (N  =  217) 


Percentile 

Number  of  Attempts 

Minimum 

0 

5‘^ 

16 

25'” 

28 

50'” 

33 

75'” 

36 

95'” 

38 

Maximum 

40 

Mean 

-  30.77 

SD 

7.29 
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Table  3.3.20.  Distribution  of  Number  of  Inappropriate  Attempts  to  Place 
a  Box  in  tbe  Loading  Area  on  the  Letter  Factory  Test  (Form  B)  (N  =  217) 


Percentile 

Number  of  Attempts 

Minimum 

0 

S* 

0 

25* 

2 

50* 

5 

75* 

13 

95* 

54 

Maximum 

135 

Mean 

13.09 

SD 

20.73 

Table  3.3.21.  Tests  of  Performance  Differences  Between  LFT  and  Retest  LFT  (N  =  184) 


LFT  Mean  Score 

Variable 

Test 

Retest 

R 

t-value 

Planning  &  Thinking  Ahead 

20.72 

22.42 

.763 

3.25’** 

Situational  Awareness 

4.70 

4.75 

.508 

0.30 

"><.001 


Table  33.22.  Distribution  of  Test  Completion  Times  for  the 
Letter  Factory  Test  (N  =  405) 


Percentile 

Test  Completion  Time 
(Minutes) 

Minimum 

31.0 

5* 

41.0 

25* 

46.4 

50* 

50.9 

75* 

55.8 

95* 

64.9 

Maximum 

107.3 

Mean 

51.77 

SD 

7.63 
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Table  3.3.23.  Proposed  Sequence  Length  and  Number  of  Situational  Awareness  Items  for  the 
Letter  Factory  Test 


Number  SA  Items 

Sequence  Length 
(Minutes) 

Number 

Level  1 

Level  2 

Level  3 

0.5 

6 

24 

2.5 

12 

24 

2 

Note.  These  recommended  sequences  would  provide  a  test  time  of  45  minutes  (not  including  instructions) 
based  on  our  experience  on  the  current  LFT. 


Table  3.3.24.  Distribution  of  Number  Correct  Scores  on  the  Scan  Test 
(N  =  429) 


Percentile 

Number  Correct  Scores 

Minimum 

22 

5“’ 

77 

25“’ 

120 

50“’ 

128 

75* 

136 

95* 

146 

Maximum 

158 

Mean 

123.73 

SD 

21.78 
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Table  3.3.25.  Scanning:  Reliability  Analyses  on  the  Scan  Test _ 

With  Change  Items®  Without  Change  Items®  Without  Any  Part  of  Change  Items 


Table  3.3.26.  Distribution  of  Test  Completion  Times  for  the  Scan  Test  (N  =  429) 


Percentile 

Test  Completion  Time  (Minutes) 

Minimum 

13.57 

S’” 

14.45 

25“’ 

15.58 

50'“ 

16.85 

75'“ 

17.93 

95'“ 

19.87 

Maximum 

25.50 

Mean 

16.92 

SD 

1.75 

Table  3.3.27.  Reliability  Analyses  on  the  Three  Parts  of  the  Planes  Test 


Test  Section 

No.  Items 

No.  Examinees’’ 

No.  Items  With 
Item-Total 
Correlations 
<.100 

Alpha 

Part  1 

48 

297 

8 

.76 

Part  2 

48 

314 

0 

.84 

Part  3 

96 

103 

24 

.80 

‘‘This  is  the  number  meeting  all  inclusion  criteria  and  having 

data  for  all  items. 

Table  3.3.28.  Distribution  of  Test  Completion  Times  for  the  Planes  Test 


Percentile 

Test  Completion  Time  (Minutes) 

Minimum 

14.1 

5'“ 

20.0 

25'“ 

25.0 

50'“ 

28.3 

75'“ 

30.6 

95'“ 

34.6 

Maximum 

39.5 

Mean 

27.73 

SD 

4.42 
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TWACXY  T1-T3  2.86  0.33  1.77  52.78  48.20  0.52  0.69  0.77  0.81  0.85  0.87  0.88  0.90 

T2-T3  0.87  1.11  0.00  59.61  41.34  0.59  0.74  0.81  0.85  0.88  0.90  0.91  0.92 


Correlation  with  Retest  Measure:  Correlation  with  Retest  Measure: 

Composite  SEPSKLL  PRCSKLL  EFFNCYA  EFFNCYB  Composite  PRACC4  PRSPD4  TWACC4 
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Table  3.3.31.  Time  Distributions  for  Current  Tests 

(Based  on  Cases  Completing  All  Trials) _ 


Test 

Instruction  Time 

Total  Time 

Median 

95"’ 

Median 

95'" 

AT 

2.5 

5.5 

9.5 

88.6 

91.7 

97.0 

TW 

1.1 

2.8 

4.8 

20.8 

24.8 

29.9 
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APPENDIX  A 

AT-SAT  Prepilot  Item  Analyses:  AM  (Applied  Math)  Test 
Items  That  Have  Been  Deleted 


AT-SAT  PREPILOT  ITEM  ANALYSES:  AM  (Applied  Math)  Test 
ITEMS  THAT  HAVE  BEEN  DELETED 
Run  03-14-1997, 09:24:05 


TOTAL  CASES  PROCESSED:  358 
FEMALES:  36 

BLACKS:  34 

HISPANICS:  41 


ITEM:  am_r008  KEY:  1  NO.  OF  OPTIONS:  4 
CORRECT  OMIT -REACH  INVLD  12  3  4 

PCT:  22.1  0.0  0.0  0.0  22.1  35.5  36.6  5.9 
BIS:  0.043  0.000  0.000  0.000  0.043  0.112-0.065-0.433 
DIF:  Female  Black  Hisp.  Low  Discrim 
%Cor  22.2  20.6  31.7  Check  Opt  2 
MnP  17.88  19.14  18.92 
BIS  0.16  0.10  -0.11  Too  Hard 
MHchi  0.00  0.01  0.96 
MHdlt  0.25  0.19  1.11 
CRIT:  AM  AM  Computed 
PrePilot  N  358  Crit  Mean:  20.94  S.D.:  7.51 


ITEM:  am_r012  KEY:  2  NO.  OF  OPTIONS:  4 
CORRECT  OMIT -REACH  INVLD  12  3  4 

PCT:  42.7  0.6  0.0  0.0  23.7  42.7  20.9  12.0 
BIS:  0.257  -0.070  0.000  0.000  0.055  0.257  -0.339  -0.278 
DIF:  Female  Black  Hisp.  Low  Discrim 
%  Cor  33.3  29.4  31.7  Check  Opt  1 
MnP  18.00  22.60  18.46 
BIS  0.24  0.60  -0.19 
MHchi  0.31  2.22  0.67 
MHdlt  -0.73  -1.71  -0.83 
CRIT:  AM  AM  Computed 
PrePilot  N  358  Crit  Mean:  20.94  S.D.:  7.51 


ITEM:  am_r013  KEY:  1  NO.  OF  OPTIONS:  4 
CORRECT  OMIT -REACH  INVLD  12  3  4 

PCT:  26.8  0.0  0.0  0.0  26.8  26.0  22.9  24.3 
BIS:  0.483  0.000  0.000  0.000  0.483  -0.278  -0.469  0.064 
DIF:  Female  Black  Hisp. 

%Cor  11.1  14.7  29.3  Check  Opt  4 

MnP  17.25  24.00  21.83 

BIS  0.05  0.55  0.36  Too  Hard 

MHchi  0.33  0.50  1.42 

MHdlt  -1.19  -1.19  1.39 

CRIT:  AM  AM  Computed 

PrePilot  N  358  Crit  Mean:  20.94  S.D.:  7.51 


AT-SAT  PREPILOT  ITEM  ANALYSES:  AM  (Applied  Math)  Test 
ITEMS  THAT  HAVE  BEEN  DELETED 
Run  03-14-1997,09:24:05 


A-2 


ITEM:  am_r016  KEY:  4  NO.  OF  OPTIONS:  4 
CORRECT  OMIT -REACH  INVLD  12  3  4 

PCT:  16.8  1.4  0.0  0.0  20.1  38.8  22.9  16.8 
BIS:  0.188  0.028  0.000  0.000-0.259  0.125-0.178  0.188 
DIF:  Female  Black  Hisp.  Low  Discrim 
%Cor  19.4  5.9  24.4  Check  Opt  2 
MnP  14.57  19.00  20.80 
BIS  -0.44  0.05  0.18  Too  Hard 
MHchi  0.15  1.68  0.69 
MHdlt  0.76  -2.56  1.19 
CRIT:  AM  AM  Computed 
PrePilot  N  358  CritMean:  20.94  S.D.:  7.51 


ITEM:  am_r021  KEY:  1  NO.  OF  OPTIONS:  4 
CORRECT  OMIT -REACH  INVLD  12  3  4 

PCT:  14.2  1.4  0.0  0.0  14.2  23.2  20.7  40.5 
BIS:  0.320  0.087  0.000  0.000  0.320  -0.299  -0.265  0.131 
DIF:  Female  Black  Hisp. 

%Cor  8.3  11.8  2.4  Check  Opt  4 

MnP  16.67  20.00  18.00 

BIS  -0.02  0.15  -0.16  Too  Hard 

MHchi  0.00  0.02  2.31 

MHdlt  -0.49  0.20  -3.64 

CRIT:  AM  AM  Computed 

PrePilot  N  358  CritMean:  20.94  S.D.:  7.51 


ITEM:  am_r022  KEY:  4  NO.  OF  OPTIONS;  4 
CORRECT  OMIT -REACH  INVLD  12  3  4 

PCT:  14.0  1.7  0.0  0.0  21.5  31.6  31.3  14.0 
BIS:  0.156  0.060  0.000  0.000  0.002-0.085-0.051  0.156 
DIF:  Female  Black  Hisp.  Low  Discrim 

%Cor  16.7  17.6  12.2  Check  Opt  1 

MnP  16.33  19.00  12.80 

BIS  -0.09  0.08  -0.92  Too  Hard 
MHchi  0.55  0.20  0.01 

MHdlt  1.28  0.81  0.23 

CRIT;  AM  AM  Computed 
PrePilot  N  358  CritMean:  20.94  S.D.:  7.51 


□ 


AT-SAT  PREPILOT  ITEM  ANALYSES:  AM  (Applied  Math)  Test 
ITEMS  THAT  HAVE  BEEN  DELETED 
Run  03-14-1997,09:24:05 


ITEM:  am_r025  KEY:  2  NO.  OF  OPTIONS:  4 
CORRECT  OMIT -REACH  INVLD  12  3  4 

PCT:  29.1  1.1  0.3  0.0  33.2  29.1  21.5  14.8 

BIS:  0.274-0.271  -0.464  0.000  0.017  0.274-0.308-0.144 
DIF:  Female  Black  Hisp.  Low  Discrim 
%  Cor  27.8  26.5  24.4  Check  Opt  1 
MnP  19.00  20.22  19.60 
BIS  0.37  0.25  0.02  Too  Hard 
MHchi  0.04  0.00  0.02 
MHdlt  0.45  0.19  0.05 
CRIT:  AM  AM  Computed 
PrePilot  N  358  Crit  Mean:  20.94  S.D.:  7.51 


ITEM:  am_r029  KEY:  2  NO.  OF  OPTIONS:  4 
CORRECT  OMIT -REACH  INVLD  12  3  4 

PCT:  41.1  0.8  1.7  0.0  16.5  41.1  24.6  15.4 

BIS:  0.246-0.098-0.491  0.000-0.208  0.246-0.163-0.007 
DIF:  Female  Black  Hisp.  Low  Discrim 
%Cor  47.2  41.2  26.8 
MnP  17.76  18.14  18.45 
BIS  0.27  -0.03  -0.18 
MHchi  2.53  0.07  2.03 
MHdlt  1.68  0.40  -1.47 
CRIT:  AM  AM  Computed 
PrePilot  N  358  Crit  Mean:  20.94  S.D.:  7.51 


ITEM:  am_r032  KEY:  3  NO.  OF  OPTIONS:  4 
CORRECT  OMIT -REACH  INVLD  12  3  4 

PCT:  33.2  1.1  2.0  0.0  23.2  25.4  33.2  15.1 
BIS:  0.192  0.208-0.450  0.000-0.103-0.095  0.192-0.088 
DIF:  Female  Black  Hisp.  Low  Discrim 
%Cor  36.1  26.5  24.4 
MnP  17.00  16.56  19.20 
BIS  0.04  -0.34  -0.05 
MHchi  0.25  0.22  0.20 
MHdlt  0.63  -0.63  -0.59 
CRIT:  AM  AM  Computed 
PrePilot  N  358  Crit  Mean:  20.94  S.D.:  7.51 


□ 


A-4 


AT-SAT  PREPILOT  ITEM  ANALYSES:  AM  (Applied  Math)  Test 
ITEMS  THAT  HAVE  BEEN  DELETED 
Run  03-14-1997, 09:24:05 


ITEM:  am_r033  KEY:  2  NO.  OF  OPTIONS:  4 
CORRECT  OMIT -REACH  INVLD  12  3  4 

PCT:  37.7  1.1  2.2  0.0  14.5  37.7  30.2  14.2 
BIS:  0.177  -0.035  -0.450  0.000  -0.134  0.177  -0.038  -0.109 
DIF:  Female  Black  Hisp.  Low  Discrim 
%Cor  30.6  35.3  39.0 
MnP  18.82  18.58  20.63 
BIS  0.37  0.05  0.21 
MHchi  0.04  0.02  0.02 
MHdlt  -0.36  -0.06  0.25 
CRIT:  AM  AM  Computed 
PrePilot  N  358  CritMean:  20.94  S.D.:  7.51 


ITEM:  am_r034  KEY:  4  NO.  OF  OPTIONS:  4 
CORRECT  OMIT  -REACH  INVLD  12  3  4 

PCT:  23.5  1.7  2.5  0.0  18.2  27.1  27.1  23.5 
BIS:  0.359  0.003-0.317  0.000-0.206-0.094-0.133  0.359 
DIF:  Female  Black  Hisp. 

%Cor  19.4  20.6  12.2 

MnP  16.14  19.57  24.20 

BIS  -0.13  0.14  0.49  Too  Hard 

MHchi  0.01  0.18  2.04 

MHdlt  0.44  0.73  -2.27 

CRIT:  AM  AM  Computed 

PrePilot  N  358  CritMean:  20.94  S.D.:  7.51 


ITEM:  am_r042  KEY:  2  NO.  OF  OPTIONS:  4 
CORRECT  OMIT  -REACH  INVLD  12  3  4 

PCT:  41.3  0.0  6.1  0.0  18.7  41.3  22.1  11.7 
BIS:  0.228  0.000-0.101  0.000-0.134  0.228-0.094-0.212 
DIF:  Female  Black  Hisp.  Low  Discrim 
%Cor  25.0  26.5  39.0 
MnP  16.89  20.56  20.19 
BIS  0.01  0.29  0.13 
MHchi  1.45  1.58  0.00  High  Omits 
MHdlt  -1.41  -1.44  -0.16 
CRIT:  AM  AM  Computed 
PrePilot  N  358  CritMean:  20.94  S.D.:  7.51 


□ 


A-5 


AT-SAT  PREPILOT  ITEM  ANALYSES:  AM  (Applied  Math)  Test 
ITEMS  THAT  HAVE  BEEN  DELETED 
Run  03-14-1997, 09:24:05 


ITEM:  am_r043  KEY:  3  NO.  OF  OPTIONS:  4 
CORRECT  OMIT -REACH  INVLD  12  3  4 

PCT:  30.7  0.6  6.4  0.0  22.6  23.7  30.7  15.9 
BIS:  0.130  0.405-0.080  0.000-0.102-0.203  0.130  0.105 
DIF:  Female  Black  Hisp.  Low  Discrim 
%Cor  22.2  32.4  29.3  Check  Opt  4 
MnP  18.38  21.55  19.17 
BIS  0.23  0.49  -0.06 
MHchi  0.39  0.00  0.04  High  Omits 
MHdlt  -0.82  0.19  -0.34 
CRIT:  AM  AM  Computed 
PrePilot  N  358  Crit  Mean:  20.94  S.D.:  7.51 


ITEM:  am_r044  KEY:  2  NO.  OF  OPTIONS:  4 
CORRECT  OMIT -REACH  INVLD  12  3  4 

PCT:  31.3  0.8  6.7  0.0  13.7  31.3  30.7  16.8 
BIS:  0.165  0.322-0.050  0.000-0.229  0.165  -0.130  0.042 
DIF:  Female  Black  Hisp.  Low  Discrim 
%Cor  25.0  29.4  24.4  Check  Opt  4 
MnP  19.67  19.20  20.50 
BIS  0.45  0.13  0.14 
MHchi  0.17  0.01  0.95  High  Omits 
MHdlt  -0.65  -0.28  -1.18 
CRIT:  AM  AM  Computed 
PrePilot  N  358  Crit  Mean:  20.94  S.D.:  7.51 


ITEM:  am_r046  KEY:  3  NO.  OF  OPTIONS:  4 
CORRECT  OMIT  -REACH  INVLD  12  3  4 

PCT:  24.6  0.6  6.7  0.0  22.3  30.4  24.6  15.4 
BIS:  0.103  0.405-0.050  0.000-0.183  0.030  0.103-0.025 
DIF:  Female  Black  Hisp.  Low  Di.scrim 
%Cor  27.8  29.4  31.7  Check  Opt  2 
MnP  18.60  20.80  18.85 
BIS  0.30  0.35  -0.12  Too  Hard 
MHchi  0.40  0.18  2.08  High  Omits 
MHdlt  0.90  0.63  1.53 
CRIT:  AM  AM  Computed 
PrePilot  N  358  Crit  Mean:  20.94  S.D.:  7.51 


□ 


AT-SAT  PREPILOT  ITEM  ANALYSES:  AM  (Applied  Math)  Test 
ITEMS  THAT  HAVE  BEEN  DELETED 
Run  03-14-1997, 09:24:05 


ITEM:  am_r048  KEY:  3  NO.  OF  OPTIONS:  4 
CORRECT  OMIT -REACH  INVLD  12  3  4 

PCT:  27.9  0.6  8.4  0.0  23.7  25.7  27.9  13.7 
BIS:  0.170  0.367  0.012  0.000-0.080-0.188  0.170  0.005 
DIF:  Female  Black  Hisp.  Low  Discrim 
%Cor  27.8  23.5  24.4  Check  Opt  4 
MnP  18.70  19.75  21.20 
BIS  0.32  0.18  0.24  Too  Hard 
MHchi  0.02  0.02  0.91  High  Omits 
MHdlt  0.39  -0.41  -1.18 
CRIT:  AM  AM  Computed 
PrePilot  N  358  CritMean:  20.94  S.D.:  7.51 


ITEM:  am_r050  KEY:  2  NO.  OF  OPTIONS:  4 
CORRECT  OMIT  -REACH  INVLD  12  3  4 

PCT:  27.7  0.3  9.5  0.0  18.2  27.7  29.6  14.8 
BIS:  0.255  0.594  0.052  0.000-0.173  0.255-0.057-0.302 
DIF:  Female  Black  Hisp.  Low  Discrim 
%Cor  25.0  32.4  22.0 
MnP  16.44  17.18  17.78 
BIS  -0.08  -0.23  -0.28  Too  Hard 
MHchi  0.03  0.52  0.15  High  Omits 
MHdlt  0.41  0.84  -0.58 
CRIT:  AM  AM  Computed 
PrePilot  N  358  CritMean:  20.94  S.D.:  7.51 


ITEM:  am_r052  KEY:  2  NO.  OF  OPTIONS:  4 
CORRECT  OMIT  -REACH  INVLD  12  3  4 

PCT:  29.6  0.0  10.1  0.0  18.7  29.6  24.9  16.8 
BIS:  0.172  0.000  0.081  0.000-0.082  0.172-0.012-0.300 
DIF:  Female  Black  Hisp.  Low  Discrim 
%Cor  30.6  26.5  36.6 
MnP  16.27  17.33  22.73 
BIS  -0.12  -0.18  0.57  Too  Hard 
MHchi  0.17  0.00  0.12  High  Omits 
MHdlt  0.60  -0.17  0.50 
CRIT:  AM  AM  Computed 
PrePilot  N  358  CritMean:  20.94  S.D.:  7.51 


□ 


AT-SAT  PREPILOT  ITEM  ANALYSES:  DI  (Dials)  Test 
ITEMS  THAT  HAVE  BEEN  DELETED 
Run  03-14-1997,  12:15:15 


TOTAL  CASES  PROCESSED:  441 
FEMALES:  48 

BLACKS:  46 

HISPANICS:  44 


ITEM:  di_r001  KEY:  2  NO.  OF  OPTIONS:  5 

CORRECT  OMIT -REACH  INVLD  1  2  3  4  5 

PCT:  93.2  0.2  0.0  0.0  0.9  93.2  3.9  1.1  0.7 
BIS:  0.196-0.245  0.000  0.000-0.362  0.196-0.025-0.333-0.207 
DIF:  Female  Black  Hisp.  Low  Discrim 
%Cor  91.7  91.3  97.7 
MnP  43.32  39.14  45.44 
BIS  0.15  0.47  0.30  Too  Easy 
MHchi  0.01  0.01  0.40 
MHdlt  0.29  0.75  2.45 
CRIT:  DI  (Dial  DI  Computed 
PrePilot  N  441  Crit  Mean:  44.47  S.D.:  7.24 


ITEM:  di_r002  KEY:  5  NO.  OF  OPTIONS:  5 

CORRECT  OMIT -REACH  INVLD  1  2  3  4  5 

PCT:  90.9  0.0  0.0  0.0  0.0  1.1  4.1  3.9  90.9 
BIS:  0.219  0.000  0.000  0.000  0.000-0.212-0.134-0.200  0.219 
DIF:  Female  Black  Hisp.  Low  Di.scrim 
%  Cor  81.3  82.6  86.4  Check  Opt  1 
MnP  44.49  39.03  45.92 
BIS  0.50  0.21  0.31  Too  Easy 
MHchi  2.61  0.96  0.52 
MHdlt  -1.92  -1.51  -1.27 
CRIT:  DI  (Dial  DI  Computed 
PrePilot  N  441  Crit  Mean:  44.47  S.D.:  7.24 


ITEM:  di_r003  KEY:  4  NO.  OF  OPTIONS:  5 

CORRECT  OMIT -REACH  INVLD  1  2  3  4  5 

PCT:  97.5  0.0  0.0  0.0  0.0  0.2  1.4  97.5  0.9 

BIS:  0.220  0.000  0.000  0.000  0.000  0.715-0.314  0.220-0.166 
DIF:  Female  Black  Hisp.  Low  Di.scrim 
%Cor  93.8  95.7  97.7  Check  Opt  2 
MnP  42.96  38.64  45.28 
BIS  -0.19  0.36  -0.08  Too  Easy 
MHchi  1.68  0.08  0.25 
MHdlt  -2.73  -1.88  0.09 
CRIT :  DI  (Dial  DI  Computed 
PrePilot  N  441  Crit  Mean:  44.47  S.D.:  7.24 
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AT-SAT  PREPILOT  ITEM  ANALYSES:  DI  (Dials)  Test 
ITEMS  THAT  HAVE  BEEN  DELETED 
Run  03-14-1997, 12:15:15 


ITEM:  di_r004  KEY:  1  NO.  OF  OPTIONS:  5 

CORRECT  OMIT -REACH  INVLD  1  2  3  4  5 

PCT:  93.4  0.0  0.0  0.0  93.4  0.0  6.6  0.0  0.0 
BIS:  0.019  0.000  0.000  0.000  0.019  0.000  -0.019  0.000  0.000 
DIF:  Female  Black  Hisp.  Low  Discrim 
%Cor  100.0  95.7  97.7  Check  Opt  2 
MnP  43.08  38.18  45.44 
BIS  0.00  -0.15  0.30  Too  Easy 
MHchi  2.46  0.07  0.64 
MHdlt  0.00  0.26  2.51 
CRIT:  DI  (Dial  DI  Computed 
PrePilot  N  441  CritMean:  44.47  S.D.:  7.24 


ITEM:  di_r005  KEY:  3  NO.  OF  OPTIONS:  5 

CORRECT  OMIT  -REACH  INVLD  1  2  3  4  5 

PCT:  79.1  0.0  0.0  0.0  3.6  12.9  79.1  3.2  1.1 
BIS:  0.220  0.000  0.000  0.000-0.261  -0.124  0.220-0.096-0.067 
DIF:  Female  Black  Hisp.  Low  Discrim 
%Cor  70.8  84.8  77.3 
MnP  44.12  40.15  45.65 
BIS  0.27  0.60  0.13 
MHchi  0.52  2.60  0.08 
MHdlt  -0.76  2.48  -0.44 
CRIT:  DI  (Dial  DI  Computed 
PrePilot  N  441  CritMean:  44.47  S.D.:  7.24 


ITEM:  di_r006  KEY:  3  NO.  OF  OPTIONS:  5 

CORRECT  OMIT -REACH  INVLD  1  2  3  4  5 

PCT:  98.9  0.0  0.0  0.0  0.0  0.9  98.9  0.2  0.0 
BIS:  0.180  0.000  0.000  0.000  0.000-0.038  0.180-0.606  0.000 
DIF:  Female  Black  Hisp.  Low  Discrim 
%Cor  100.0  97.8  100.0  Check  Opt  1 
MnP  43.08  38.56  45.30 
BIS  0.00  0.51  0.00  Too  Easy 
MHchi  0.08  0.24  0.05 
MHdlt  0.00  0.30  0.00 
CRIT:  DI(Dial  DI  Computed 
PrePilot  N  441  CritMean:  44.47  S.D.:  7.24 
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AT-SAT  PREPILOT  ITEM  ANALYSES:  DI  (Dials)  Test 
ITEMS  THAT  HAVE  BEEN  DELETED 
Run  03-14-1997, 12:15:15 


ITEM:  di_r013  KEY:  5  NO.  OF  OPTIONS:  5 

CORRECT  OMIT -REACH  INVLD  1  2  3  4  5 

PCT:  42.6  0.2  0.0  0.0  0.2  11.8  8.4  36.7  42.6 
BIS:  0.070-0.377  0.000  0.000  0.050-0.048-0.117  0.049  0.070 
DIF:  Female  Black  Hisp.  Low  Discrim 
%Cor  41.7  45.7  45.5  Check  Opt  4 
MnP  44.00  39.29  44.85 
BIS  0.15  0.11  -0.08 
MHchi  0.01  0.14  0.08 
MHdlt  0.03  0.48  0.35 
CRIT:  DI  (Dial  DI  Computed 
PrePilot  N  441  CritMean:  44.47  S.D.:  7.24 


ITEM:  di_r014  KEY:  4  NO.  OF  OPTIONS:  5 

CORRECT  OMIT -REACH  INVLD  1  2  3  4  5 

PCT:  42.0  0.0  0.0  0.0  0.9  1.1  55.6  42.0  0.5 
BIS:  0.164  0.000  0.000  0.000  0.003-0.526-0.076  0.164-0.482 
DIF:  Female  Black  Hisp.  Low  Discrim 
%Cor  43.8  21.7  25.0  Check  Opt  1 
Mn  P  45.43  39.60  47.64 
BIS  0.38  0.11  0.37 
MHchi  0.13  3.86  4.63 
MHdlt  0.40  -1.90* -2.07*  Possible  DIF 
CRIT:  DI  (Dial  DI  Computed 
PrePilot  N  441  CritMean:  44.47  S.D.:  7.24 


ITEM:  di_r016  KEY:  1  NO.  OF  OPTIONS:  5 

CORRECT  OMIT -REACH  INVLD  1  2  3  4  5 

PCT:  86.4  0.0  0.0  0.0  86.4  11.8  0.9  0.2  0.7 
BIS:  0.061  0.000  0.000  0.000  0.061  0.111  -0.450-0.180-0.610 
DIF:  Female  Black  Hisp.  Low  Discrim 
%Cor  89.6  84.8  84.1  Check  Opt  2 
MnP  44.07  39.31  45.00 
BIS  0.53  0.33  -0.23 
MHchi  0.62  0.11  0.05 
MHdlt  1.29  0.63  -0.48 
CRIT:  DI  (Dial  DI  Computed 
PrePilot  N  441  CritMean:  44.47  S.D.:  7.24 


AT-SAT  PREPILOT  ITEM  ANALYSES:  DI  (Dials)  Test 
ITEMS  THAT  HAVE  BEEN  DELETED 
Run  03-14-1997, 12:15:15 


ITEM:  di_r034  KEY:  1  NO.  OF  OPTIONS:  5 

CORRECT  OMIT -REACH  INVLD  1  2  3  4  5 

PCT:  48.1  0.0  1.4  0.0  48.1  1.1  42.9  2.3  4.3 

BIS:  0.297  0.000-0.931  0.000  0.297-0.606  0.041  -0.151-0.352 
DIF:  Female  Black  Hisp.  Low  Discrim 
%Cor  56.3  34.8  52.3  Check  Opt  3 
Mn  P  43.96  43.38  46.22 
BIS  0.17  0.51  0.20 
MHchi  2.71  0.04  0.06 
MHdlt  1.44  -0.34  0.32 
CRIT:  DI  (Dial  DI  Computed 
PrePilot  N  441  CritMean:  44.47  S.D.:  7.24 


ITEM:  di_r037  KEY:  1  NO.  OF  OPTIONS:  5 

CORRECT  OMIT  -REACH  INVLD  1  2  3  4  5 

PCT:  53.3  0.0  1.8  0.0  53.3  39.2  3.2  0.7  1.8 
BIS:  0.254  0.000-0.936  0.000  0.254-0.004-0.186-0.106-0.296 
DIF:  Female  Black  Hisp.  Low  Discrim 
%Cor  50.0  43.5  43.2 
MnP  43.54  42.95  45.53 
BIS  0.08  0.51  0.04 
MHchi  0.05  0.03  2.04 
MHdlt  -0.28  -0.02  -1.24 
CRIT:  DI  (Dial  DI  Computed 
PrePilot  N  441  CritMean:  44.47  S.D.:  7.24 


ITEM:  di_r040  KEY:  2  NO.  OF  OPTIONS:  5 

CORRECT  OMIT -REACH  INVLD  1  2  3  4  5 

PCT:  40.8  0.0  2.3  0.0  50.3  40.8  3.4  1.8  1.4 
BIS:  0.156  0.000-0.918  0.000  0.229  0.156-0.250-0.317-0.561 
DIF:  Female  Black  Hisp.  Low  Discrim 
%Cor  50.0  39.1  34.1  Check  Opt  1 
MnP  42.88  42.28  47.67 
BIS  0.04  0.42  0.42 
MHchi  0.76  0.06  1.25 
MHdlt  0.82  0.32  -1.11 
CRIT:  DI  (Dial  DI  Computed 
PrePilot  N  441  CritMean:  44.47  S.D.:  7.24 
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AT-SAT  PREPILOT  ITEM  ANALYSES:  DI  (Dials)  Test 
ITEMS  THAT  HAVE  BEEN  DELETED 
Run  03-14-1997,  12:15:15 


ITEM:  di_r047  KEY:  5  NO.  OF  OPTIONS:  5 

CORRECT  OMIT  -REACH  INVLD  1  2  3  4  5 

PCT:  62.6  0.5  3.9  0.0  0.2  2.5  19.5  10.9  62.6 
BIS:  0.468  -0.088  -0.841  0.000  -0.508  -0.244  0.041  -0.266  0.468 
DIF:  Female  Black  Hisp. 

%Cor  60.4  45.7  63.6  Check  Opt  3 

MnP  45.90  43.76  45.54 

BIS  0.60  0.61  0.06 

MHchi  0.02  0.21  0.63 

MHdlt  0.03  -0.61  -0.80 

CRIT:  DI  (Dial  DI  Computed 

PrePilot  N  441  CritMean:  44.47  S.D.:  7.24 
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APPENDIX  B 

Descriptive  Statistics,  Internal  Consistency  Reliabilities, 
Intercorrelations,  and  Factor  Analysis  Results  for 
Experience  Questionnaire  Scales 
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Descriptive  Statistics,  Internal  Consistency  Reliabilities, 
Intercorrelations,  and  Factor  Analysis  Results  for 
Experience  Questionnaire  Scales 

Descriptive  Statistics  for  Experience  Questionnaire  Scales 


Variable 

Mean 

Std 

Dev 

TINTENSE 

64 

.  17 

12 

.  66 

TCOMPOSE 

64 

.28 

12 

.  11 

TDECIDE 

62 

.06 

12 

.  13 

TEXECUTE 

71 

.  05 

12 

.01 

TTAKECHG 

73 

.11 

11 

.75 

TCONCEN 

66 

.  93 

13 

.46 

TELEX 

68 

.  02 

12 

.27 

TSELFAW 

72 

.43 

9 

.26 

TCOOP 

75, 

.73 

11 

.  18 

TSUSTAIN 

67  , 

.  87 

14 

.  73 

TSELFCON 

72. 

,51 

12 

.  77 

TSELFMON 

71 . 

.  57 

9 

.26 

TTASK 

70. 

81 

11 

.72 

TUV 

51 . 

85 

13 

.  18 

TINTTOL 

70  . 

26 

12 

.50 

TCONSIST 

64  . 

08 

11 

.29 

Valid 


Minimum 

Maximum 

N 

30 . 0000 

100 . 0000 

270 

30.6667 

100 . 0000 

258 

20 . 0000 

97.3333 

256 

24.0000 

100 . 0000 

268 

25.3333 

100.0000 

258 

25.3333 

100 . 0000 

257 

20.0000 

100.0000 

257 

36.0000 

100.0000 

269 

20.0000 

100.0000 

257 

30 . 0000 

100 . 0000 

269 

36.0000 

100.0000 

268 

36.0000 

100 . 0000 

268 

25.3333 

100 . 0000 

257 

22 . 0000 

100 . 0000 

267 

27 . 2727 

100 . 0000 

257 

32.7273 

100 . 0000 

260 

Internal  Consistency  Reliability  Analyses  (including  item-level  descriptive 

statistics) 


Reliability  of  EQ  Scales  (includes  item-level  descriptive  statistics) 


RELIABILI 


1 . 

EQl 

2  . 

EQ147 

3  . 

EQ132 

4  . 

EQ115 

5  . 

EQIOO 

6  . 

EQ85 

7  . 

EQ68 

8  . 

EQ51 

9  . 

EQ35 

10  . 

EQ18 

TY  ANALYS 

Mean 

3 .8906 
3.0469 
2.3984 
2.8828 
3.8203 
2 . 6758 
3.8438 
2.9453 
2.5469 
4 . 0391 


IS  -  S  C  A  L 

Std  Dev 

.  9266 
1.2101 
1 . 1536 
1 . 3083 
1 . 0511 
1.2112 
1 . 1509 
1.2323 
1 . 1051 
.  9448 


:  (INTENSE) 

Cases 

256 . 0 
256.0 
256 . 0 
256 . 0 
256 . 0 
256 . 0 
256 . 0 
256 . 0 
256 . 0 
256 . 0 
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N  of 

Statistics  for  Mean  Variance  Std  Dev  Variables 


SCALE 

32 . 0898 

40.1762 

6.3385 

10 

Item-total 

Statistics 

Scale 

Scale 

Corrected 

Mean 

Variance 

Item- 

Alpha 

if  Item 

if  Item 

Total 

if  Item 

Deleted 

Deleted 

Correlation 

Deleted 

EQl 

28 . 1992 

35.7288 

.3240 

.  7460 

EQ147 

29.0430 

32.8570 

.4220 

.7336 

EQ132 

29 . 6914 

33.3436 

.4130 

.  7348 

EQ115 

29.2070 

31.7805 

.4531 

.  7291 

EQIOO 

28.2695 

34 . 0957 

.4053 

.  7360 

EQ85 

29.4141 

32.2749 

.4675 

.  7267 

EQ68 

28.2461 

33.4176 

.  4084 

.  7355 

EQ51 

29 . 1445 

31 . 0496 

.5540 

.  7128 

EQ35 

29.5430 

34.8452 

.3150 

.  7481 

EQ18 

28 . 0508 

34 . 9111 

.3916 

.  7382 

Reliability 

Coefficients 

N  of  Cases 

=  256.0 

N  of  Items  = 

10 

Alpha  =  .7545 


E  L 

lABILITY 

ANALYSIS 

SC 

ALE  (C 

Mean 

Std  Dev 

Cases 

1 . 

EQ2 

4.2969 

.8389 

256 . 0 

2  . 

EQ19 

2.5742 

1.2313 

256.0 

3  . 

EQ36 

3 . 1406 

1.2158 

256.0 

4  . 

EQ52 

2 . 8555 

1.2889 

256 . 0 

5  . 

EQ69 

1 . 9805 

1.0533 

256 . 0 

6. 

EQ86 

3.5820 

1 . 1312 

256 . 0 

7  . 

EQ102 

3 . 0078 

1.2045 

256 . 0 

8  . 

EQ117 

3.8828 

.  9507 

256 . 0 

9. 

EQ133 

3.4570 

1.3742 

256.0 

10. 

EQ148 

2 . 9766 

1.3661 

256.0 

11 . 

EQ162 

3.1406 

1.2190 

256.0 

12  . 

EQ170 

3.0391 

1.3070 

256 . 0 

13  . 

EQ178 

3.9531 

.  9277 

256 . 0 

14  . 

EQ186 

2.4727 

1.1303 

256 . 0 

15  . 

EQ194 

3 . 8789 

.  9727 

256.0 
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N  of 


Statistics  for 

Mean 

Variance 

Std  Dev 

Variables 

SCALE 

48 . 2383 

83.0214 

9.1116 

15 

Item-total 

Statistics 

Scale 

Scale 

Corrected 

Mean 

Variance 

Item- 

Alpha 

if  Item 

if  Item 

Total 

if  Item 

Deleted 

Deleted 

Correlation 

Deleted 

EQ2 

43 . 9414 

77.4671 

.3285 

.  8068 

EQ19 

45  .  6641 

70 . 8671 

.5132 

.7940 

EQ36 

45.0977 

71.4767 

.4897 

.7958 

EQ52 

45.3828 

71 . 1548 

.4693 

.7973 

EQ69 

46.2578 

76.1999 

.  3106 

.  8080 

EQ86 

44  .  6563 

73.0265 

.4508 

.7988 

EQ102 

45.2305 

70.8996 

.  5261 

.7931 

EQ117 

44,3555 

74 . 9124 

.4378 

.8004 

EQ133 

44 . 7813 

71 . 9127 

.3956 

.  8037 

EQ148 

45.2617 

74  .  6018 

.2777 

.  8132 

EQ162 

45 . 0977 

69.3277 

.  6014 

.  7872 

EQ170 

45 .1992 

72.2151 

.4096 

.  8021 

EQ178 

44  .  2852 

74 . 1497 

.5014 

.7970 

EQ186 

45.7656 

76.4625 

.2672 

,  8113 

EQ194 

44.3594 

74 . 7409 

.4361 

.  8004 

B-4 


Reliability  Coefficients 


N  of  Cases  =  256.0  N  of  Items  =  15 

Alpha  =  .8116 


R  E  L 

lABILITY  A 

N  A  L  Y  S  I 

S  -  SC 

ALE  (D  E  C  I 

Mean 

Std  Dev 

Cases 

1  . 

EQ4 

4 . 0898 

.8091 

256.0 

2  . 

EQ21 

2 . 1406 

.  9999 

256.0 

3  . 

EQ37 

3.1523 

1 . 1733 

256.0 

4  . 

EQ53 

3 . 0234 

1.2428 

256.0 

5  . 

EQ70 

3.1680 

1 . 1911 

256.0 

6  . 

EQ87 

2 . 8984 

1 . 0762 

256.0 

7  . 

EQ103 

2 . 9141 

1.2774 

256.0 

8  . 

EQ118 

2 . 6875 

1.3180 

256 . 0 

9. 

EQ134 

3.7383 

.9527 

256.0 

10  . 

EQ149 

3 . 0977 

1 . 1420 

256.0 

11 . 

EQ164 

3.2773 

1.1603 

256.0 

12. 

EQ171 

3.1211 

1.1869 

256.0 

13. 

EQ180 

3.8047 

.8949 

256.0 

14  . 

EQ187 

2.3984 

1.3392 

256.0 

15  . 

EQ195 

3.0352 

1.2186 

256.0 

N  of 

Statistics  for  Mean 

Variance 

Std  Dev 

Variables 

SCALE  46.5469 

82.8135 

9.1002 

15 

Item-total 

Statistics 

Scale 

Scale 

Corrected 

Mean 

Variance 

Item- 

Alpha 

if  Item 

if  Item 

Total 

if  Item 

Deleted 

Deleted 

Correlation 

Deleted 

EQ4 

42 . 4570 

77 . 4491 

.3307 

.  8139 

EQ21 

44.4063 

82.1716 

-.0197 

.  8336 

EQ37 

43.3945 

70.0124 

.5818 

.  7971 

EQ53 

43.5234 

82.5014 

-.0546 

.  8417 

EQ70 

43.3789 

69.8598 

.5793 

.7971 

EQ87 

43.6484 

71 . 9936 

.5290 

.  8015 

EQ103 

43.6328 

67 . 1039 

.  6727 

.  7891 

EQ118 

43 . 8594 

68 . 7252 

.5652 

.7975 

EQ134 

42 . 8086 

75 . 6534 

.3772 

.  8113 

EQ149 

43.4492 

70.9307 

.5499 

.7996 

EQ164 

43.2695 

72.8094 

.4372 

.  8074 

EQ171 

43 . 4258 

70.7239 

.5350 

.  8004 

EQ180 

42 . 7422 

75.3921 

.4260 

.  8088 

EQ187 

44 . 1484 

71 .5622 

.4174 

.  8094 

EQ195 

43 . 5117 

69.2548 

.5953 

.7957 

B-5 


Reliability  Coefficients 


N  of  Cases  =  256.0  N  of  Items  =  15 

Alpha  =  .8183 


RELIABILITY  ANALYSIS  -  SCALE  (EXECUTE) 


Mean 

Std  Dev 

Cases 

1 . 

EQ5 

4 . 0977 

.  8780 

256 . 0 

2  . 

EQ23 

3.7891 

.8646 

256 . 0 

3. 

EQ39 

3.5508 

1 . 1871 

256.0 

4  . 

EQ56 

3.9922 

.  9372 

256.0 

5  . 

EQ71 

4 . 0469 

.  9019 

256 . 0 

6. 

EQ88 

3.3828 

1  .  4287 

256.0 

7  . 

EQ104 

2 . 6797 

1.3999 

256.0 

8  . 

EQ119 

3.2188 

1.1745 

256.0 

9. 

EQ135 

3 . 1875 

1 . 1564 

256.0 

10  . 

EQ150 

3.4961 

1 . 1479 

256 . 0 

N  of 

Statistics  for 

Mean 

Variance 

Std  Dev 

Variables 

SCALE 

35 .4414 

36.3260 

6.0271 

10 

Item-total 

Statistics 

Scale 

Scale 

Corrected 

Mean 

Variance 

Item- 

Alpha 

if  Item 

if  Item 

Total 

if  Item 

Deleted 

Deleted 

Correlation 

Deleted 

EQ5 

31 . 3438 

32 . 0461 

.3530 

.  7078 

EQ23 

31 . 6523 

31 . 3100 

.4412 

.  6972 

EQ39 

31.8906 

28 .8115 

.4791 

.  6859 

EQ56 

31 .4492 

31 . 0798 

.4180 

.  6987 

EQ71 

31 . 3945 

31 .1182 

.  4367 

.  6969 

EQ88 

32 . 0586 

29.1064 

.3359 

.  7148 

EQ104 

32 .7617 

29.4293 

.3250 

.7162 

EQ119 

32.2227 

28.2757 

.  5341 

.  6764 

EQ135 

32.2539 

30.3863 

.3610 

.  7060 

EQ150 

31 . 9453 

31 . 6990 

.2560 

.  7227 

B-6 


Reliability  Coefficients 


N  of  Cases  =  256.0 

Alpha  =  .7240 


R  E  L 

lABILITY 

A  N  A  L  y  S 

Mean 

1  . 

EQ7 

4 . 0508 

2  . 

EQ24 

2.9688 

3  . 

EQ40 

3.5625 

4  . 

EQ57 

3.8789 

5  . 

EQ72 

2.9570 

6  . 

EQ89 

3.5859 

7  . 

EQ105 

4 . 0625 

8  . 

EQ120 

3.8359 

9  . 

EQ137 

3.9258 

10  . 

EQ152 

2 . 7852 

11 . 

EQ165 

3.6836 

12  . 

EQ172 

3.9531 

13. 

EQ181 

4 . 1797 

14  . 

EQ188 

3.7109 

15  . 

EQ196 

3.7188 

Statistics  for 

Mean 

Variance 

SCALE 

54 . 8594 

78 . 0272 

Item-total 

Statistics 

Scale 

Scale 

Mean 

Variance 

if  Item 

if  Item 

Deleted 

Deleted 

EQ7 

50 . 8086 

70 . 4377 

EQ2  4 

51 .8906 

69.3841 

EQ4  0 

51.2969 

66.6174 

EQ57 

50 . 9805 

66.6310 

EQ72 

51 . 9023 

68 . 0963 

EQ89 

51.2734 

68.7171 

EQ105 

50.7969 

70 . 5468 

EQ120 

51 . 0234 

71.0112 

EQ137 

50 . 9336 

66.4387 

EQ152 

52 . 0742 

68.2180 

EQ165 

51 . 1758 

67.7925 

EQ172 

50 . 9063 

69.5127 

EQ181 

50 . 6797 

70 . 4774 

EQ188 

51 . 1484 

69.1700 

EQ196 

51 . 1406 

67.1645 

N  of  Items  =  10 


S  -  SCALE  (TAKECHG) 


Std  Dev 

Cases 

.8125 

256 . 0 

1.1878 

256 . 0 

1.1220 

256 . 0 

1.0797 

256 . 0 

1 . 1890 

256 . 0 

1.0406 

256 . 0 

.  9139 

256 . 0 

.  9270 

256 . 0 

.  9693 

256.0 

1.2973 

256.0 

.  9929 

256.0 

.  9149 

256.0 

.8207 

256 . 0 

.  9794 

256.0 

1 . 0474 

256.0 

N  of 

Std  Dev  Variables 
8.8333  15 


Corrected 

Item- 

Alpha 

Total 

if  Item 

Correlation 

Deleted 

.5081 

.  8445 

.3655 

.  8526 

.5542 

.  8409 

.5804 

.  8394 

.4340 

.  8484 

.4769 

.  8453 

.4328 

.  8475 

.3941 

.  8494 

.  6739 

.  8350 

.3792 

.  8531 

.5657 

.  8406 

.5033 

.  8442 

.4991 

.  8448 

.4848 

.  8449 

.5688 

.8402 

B-7 


Reliability 

Coefficients 

N  of  Cases  = 

256.0 

N  of  Items  =  15 

Alpha  = 

8536 

R  E 

L  I  A  B  I  L 

I  T  Y  A 

N  A  L  Y  S  I 

S  -  S 

CALE  (CONCEN) 

Mean 

Std  Dev 

Cases 

1  . 

EQ8 

3.8867 

.  9856 

256 . 0 

2  . 

EQ25 

2.5391 

1.2201 

256.0 

3. 

EQ41 

3.6797 

1 . 1745 

256 . 0 

4  . 

EQ58 

2.7695 

1.3097 

256 . 0 

5  . 

EQ73 

3.8828 

.  9951 

256.0 

6. 

EQ90 

2.8555 

1 . 3367 

256 . 0 

7  . 

EQ106 

3.4883 

1 . 0174 

256 . 0 

8  . 

EQ121 

3.6641 

1.3002 

256.0 

9. 

EQ138 

3.7383 

1.0469 

256.0 

10. 

EQ153 

3.3711 

1.2235 

256 . 0 

11 . 

EQ166 

3 . 1367 

1 . 3049 

256 . 0 

12  . 

EQ173 

3.1328 

1.3306 

256 . 0 

13. 

EQ182 

3 . 6641 

1 . 0311 

256 . 0 

14  . 

EQ189 

3.8516 

.  9948 

256 . 0 

15. 

EQ197 

2 . 5820 

1.2869 

256 . 0 

N  of 

Statistics  for 

Mean 

Variance 

Std  Dev 

Variables 

SCALE 

50 . 2422 

101 . 6823 

10 . 0838 

15 

Item-total 

Statistics 

Scale 

Scale 

Corrected 

Mean 

Variance 

Item- 

Alpha 

if  Item 

if  Item 

Total 

if  Item 

Deleted 

Deleted 

Correlation 

Deleted 

EQ8 

46 . 3555 

92.2065 

.  4493 

.  8450 

EQ25 

47 .7031 

89.3390 

.4706 

.  8437 

EQ41 

46.5625 

90.2863 

.  4488 

.  8449 

EQ58 

47 . 4727 

85.5679 

.  5943 

.  8363 

EQ73 

46.3594 

90 . 8586 

.5184 

.  8417 

EQ90 

47 . 3867 

91.2028 

.  3405 

.  8520 

EQ106 

46.7539 

91.2059 

.4858 

.  8432 

EQ121 

46.5781 

89.0998 

.  4437 

.  8456 

EQ138 

46 . 5039 

88.7843 

.  5982 

.  8375 

EQ153 

46.8711 

87 . 5794 

.5505 

.  8392 

EQ166 

47 .1055 

84 . 1810 

.  6598 

.  8322 

EQ173 

47 . 1094 

88 . 1919 

.4689 

.  8442 

EQ182 

46.5781 

89.8684 

.5499 

.  8400 

EQ189 

46 . 3906 

95.2272 

.2815 

.  8525 

EQ197 

47 . 6602 

89.1037 

.4496 

.  8452 

B-8 

Reliability  Coefficients 


N  of  Cases  =  256.0  N  of  Items  =  15 

Alpha  =  . 8519 


R  E  L 

lABILITY  A 

N  A  L  Y  S  I 

S  -  SC 

ALE  (FLEX) 

Mean 

Std  Dev 

Cases 

1  . 

EQ9 

3 . 4414 

1.3093 

256 . 0 

2  . 

EQ2  6 

3.7266 

.  9799 

256 . 0 

3. 

EQ43 

3 .8906 

.  9680 

256 . 0 

4  . 

EQ59 

3 . 0859 

1.2713 

256 . 0 

5  . 

EQ92 

3.7305 

1.0598 

256.0 

6. 

EQ107 

2.5898 

1.2174 

256 . 0 

7  . 

EQ122 

3 .8828 

.  9341 

256 . 0 

8  . 

EQ139 

3 . 1484 

1 . 1953 

256.0 

9. 

EQ154 

3 . 3867 

1.2410 

256.0 

10  . 

EQ167 

3 . 6680 

1 . 0678 

256.0 

11 . 

EQ174 

2 . 8398 

1.1349 

256.0 

12  . 

EQ183 

4 . 1016 

1.0046 

256.0 

13. 

EQ190 

2 . 7500 

1.1816 

256.0 

14  . 

EQ198 

3.3828 

1.1956 

256.0 

N  of 

Statistics  for  Mean 

Variance 

Std  Dev 

Variables 

SCALE  47.6250 

74.0000 

8.6023 

14 

Item-total 

Statistics 

Scale 

Scale 

Corrected 

Mean 

Variance 

Item- 

Alpha 

if  Item 

if  Item 

Total 

if  Item 

Deleted 

Deleted 

Correlation 

Deleted 

EQ9 

44 . 1836 

64 . 9583 

.3472 

.  8128 

EQ26 

43 .8984 

66.7661 

.3918 

.  8079 

EQ43 

43 . 7344 

68.1645 

.3065 

.  8132 

EQ5  9 

44 . 5391 

60.4926 

.  6013 

.  7910 

EQ92 

43 .8945 

64 . 7222 

.4782 

.  8019 

EQ107 

45 . 0352 

65.8537 

.3373 

.  8126 

EQ122 

43 . 7422 

67 . 0627 

.3964 

.  8077 

EQ139 

44 . 4766 

65.0818 

.3883 

.  8085 

EQ154 

44.2383 

59.7352 

.  6633 

.  7860 

EQ167 

43 . 9570 

64.2923 

.5003 

.  8003 

EQ174 

44 . 7852 

65.5341 

.3906 

.  8081 

EQ183 

43 . 5234 

66.9328 

.3685 

.  8093 

EQ190 

44 . 8750 

64 . 0941 

.  4498 

.  8038 

EQ198 

44.2422 

62.2470 

.5472 

.  7960 

B-9 


Reliability  Coefficients 


N  of  Cases  =  256.0  N  of  Items  =  14 

Alpha  =  .8159 


R  E 

L  I  A  B  I  L 

I  T  Y  A 

N  A  L  Y  S  I 

S  -  S 

CALE  (SELF 

Mean 

Std  Dev 

Cases 

1  . 

EQIO 

3.8398 

.  9629 

256 . 0 

2  . 

EQ27 

2 .4102 

1 . 3105 

256 . 0 

3  . 

EQ44 

4.2383 

.  8459 

256 . 0 

4  . 

EQ60 

4 . 1094 

1.0863 

256 . 0 

5. 

EQ77 

2.6172 

1 . 1070 

256 . 0 

6. 

EQ93 

4  . 1406 

.  9434 

256.0 

7  . 

EQ108 

3 . 8477 

1 .1998 

256.0 

8  . 

EQ123 

4 .1953 

.8360 

256.0 

9. 

EQ140 

2.6016 

1 . 1871 

256.0 

10. 

EQ155 

4.1680 

.  9070 

256.0 

N  of 

Statistics  for 

Mean 

Variance 

Std  Dev 

Variables 

SCALE 

36.1680 

21 . 8344 

4 . 6727 

10 

Item-total 

Statistics 

Scale 

Scale 

Corrected 

Mean 

Variance 

Item- 

Alpha 

if  Item 

if  Item 

Total 

if  Item 

Deleted 

Deleted 

Correlation 

Deleted 

EQIO 

32 . 3281 

18 . 9194 

.2373 

.5245 

EQ27 

33 . 7578 

18 . 0274 

.  1878 

.  5441 

EQ4  4 

31 . 9297 

18 . 7872 

.3180 

.  5076 

EQ60 

32 . 0586 

18 . 6750 

.2108 

.  5319 

EQ77 

33 . 5508 

19.0327 

.1632 

.  5457 

EQ93 

32 . 0273 

18 . 6463 

.2821 

.  5135 

EQ108 

32 . 3203 

17.5833 

.2794 

.  5116 

EQ123 

31 . 9727 

18 . 6855 

.  3390 

.  5032 

EQ140 

33 . 5664 

18 . 6387 

.  1743 

.  5446 

EQ155 

32 . 0000 

18 . 7059 

.2939 

.  5114 

B-10 


Reliability  Coefficients 


N  of  Cases  =  256.0 

Alpha  =  .5500 


R  E  L 

lABILITY  A 

N  A  L  Y  S 

Mean 

1  . 

EQll 

4 . 3047 

2. 

EQ28 

4.3828 

3  . 

EQ45 

4 . 1094 

4  . 

EQ61 

2 . 8008 

5  . 

EQ78 

4.2969 

6. 

EQ94 

3 . 9219 

7  . 

EQ109 

4 . 1484 

8  . 

EQ124 

3.8789 

9. 

EQ141 

3 . 9453 

10  . 

EQ156 

3.0664 

11 . 

EQ168 

3.3555 

12  . 

EQ175 

4.0352 

13. 

EQ184 

3.9453 

14  . 

EQ192 

3 . 8047 

15  . 

EQ199 

2 . 7734 

Statistics  for  Mean 

SCALE  56.7695 

Variance 

70 . 4055 

Item-total 

Statistics 

Scale 

Scale 

Mean 

Variance 

if  Item 

if  Item 

Deleted 

Deleted 

EQll 

52.4648 

62 . 4772 

EQ28 

52.3867 

62 . 6303 

EQ45 

52 . 6602 

62 . 0527 

EQ61 

53 . 9688 

60 . 4147 

EQ78 

52 . 4727 

62 . 0855 

EQ94 

52 . 8477 

61.0630 

EQ109 

52 . 6211 

60.7461 

EQ124 

52 . 8906 

61.3919 

EQ141 

52 . 8242 

62.0278 

EQ156 

53.7031 

61.9037 

EQ168 

53 . 4141 

60.9338 

EQ175 

52.7344 

64 . 0703 

EQ184 

52 . 8242 

62.3102 

EQ192 

52 . 9648 

62.5360 

EQ199 

53.9961 

63.8157 

N  of  Items  =  10 


S  -  SCALE  (WORKCOOP) 


Std  Dev 

Cases 

.8592 

256 . 0 

.8319 

256 . 0 

.  9680 

256.0 

1.1896 

256 . 0 

.7857 

256 . 0 

.  9547 

256.0 

.  9587 

256.0 

.  9807 

256 . 0 

.9314 

256.0 

1.2585 

256.0 

1 . 1860 

256.0 

.9915 

256.0 

.  9440 

256.0 

.9866 

256.0 

1.1356 

256.0 

N  of 

Std  Dev  Variables 
8.3908  15 


Corrected 

Item- 

Alpha 

Total 

if  Item 

Correlation 

Deleted 

.5294 

.  8279 

.5379 

.  8277 

.4863 

.  8296 

.4637 

.  8314 

.  6221 

.  8242 

.5650 

.  8252 

.5849 

.  8240 

.5239 

.  8274 

.5119 

.  8283 

.3493 

.  8403 

.4356 

.8334 

.3372 

.  8382 

.4834 

.  8298 

.4419 

.  8321 

.2921 

.  8424 

B-11 


Reliability  Coefficients 


N  of  Cases  -  256.0  N  of  Items  =  15 

Alpha  =  .8403 


RELIABILITY  ANALYSIS  -  SCALE  (SUSATTN) 


Mean 

Std  Dev 

Cases 

1 . 

EQ12 

3 . 5078 

1 . 0175 

256 . 0 

2  . 

EQ2  9 

3.5664 

1 . 1797 

256 . 0 

3  . 

EQ4  6 

3 . 9844 

1.2710 

256.0 

4  . 

EQ62 

3.1992 

1.3588 

256 . 0 

5. 

EQ79 

2 . 4844 

1.2555 

256.0 

6. 

EQ95 

3.5352 

1.2829 

256 . 0 

7  . 

EQllO 

3.8984 

1 .1904 

256.0 

8  . 

EQ126 

3.6680 

1.1458 

256.0 

9. 

EQ142 

2.8828 

1.2872 

256.0 

10  . 

EQ157 

3 . 1289 

1.2379 

256.0 

N  of 

Statistics  for 

Mean 

Variance 

Std  Dev 

Variables 

SCALE 

33 . 8555 

55.0731 

7.4211 

10 

Item- 

total  Statistics 

Scale 

Scale 

Corrected 

Mean 

Variance 

Item- 

Alpha 

if  Item 

if  Item 

Total 

if  Item 

Deleted 

Deleted 

Correlation 

Deleted 

EQ12 

30 . 3477 

47 . 7806 

.4448 

.7955 

EQ2  9 

30.2891 

45.8691 

.4889 

.  7905 

EQ4  6 

29.8711 

47 . 0774 

.3658 

.  8044 

EQ62 

30 . 6563 

42.6108 

.  5984 

.  7769 

EQ79 

31 . 3711 

43.7716 

.  5854 

.  7792 

EQ95 

30 . 3203 

45.5362 

.4558 

.7943 

EQllO 

29 . 9570 

46.0178 

.  4729 

.7922 

EQ126 

30 . 1875 

47 . 6902 

.3836 

.  8014 

EQ142 

30 . 9727 

45.1953 

.  4750 

.7921 

EQ157 

30 . 7266 

44 . 0661 

.5765 

.  7804 

B-12 


Reliability  Coefficients 


N  of  Cases  =  256.0  N  of  Items  =  10 

Alpha  =  .8079 

RELIABILITY  ANALYSIS  -  SCALE  (SELFCON) 


Mean 

Std  Dev 

Cases 

1 . 

EQ13 

3 . 5039 

.8722 

256 . 0 

2  . 

EQ31 

4 . 3164 

.8340 

256 . 0 

3  . 

EQ47 

4 . 1367 

.8727 

256 . 0 

4  . 

EQ63 

4 . 0352 

.7480 

256 . 0 

5  . 

EQ80 

3.1172 

1.2626 

256 . 0 

6  . 

EQ96 

3.2109 

1.3204 

256.0 

7  . 

EQlll 

3 . 9531 

.  9527 

256.0 

8  . 

EQ127 

3.0859 

1.3078 

256.0 

9. 

EQ143 

3.9375 

.  9182 

256.0 

10  . 

EQ158 

2.9492 

1.2991 

256.0 

N  of 

Statistics  for 

Mean 

Variance 

Std  Dev 

Variables 

SCALE 

36.2461 

40 . 9941 

6.4027 

10 

Item-total 

Statistics 

Scale 

Scale 

Corrected 

Mean 

Variance 

Item- 

Alpha 

if  Item 

if  Item 

Total 

if  Item 

Deleted 

Deleted 

Correlation 

Deleted 

EQ13 

32 . 7422 

36.5215 

.3521 

.  8015 

EQ31 

31 . 9297 

36.1205 

.4168 

.  7958 

EQ47 

32 .1094 

36.5527 

.3487 

.  8018 

EQ63 

32.2109 

37 . 0926 

.  3668 

.  8003 

EQ80 

33 . 1289 

32 . 9833 

.  4425 

.7952 

EQ96 

33 . 0352 

30.5909 

.5929 

.  7749 

EQlll 

32.2930 

34 . 1923 

.5290 

.  7843 

EQ127 

33 .1602 

30 . 4252 

.  6140 

.  7718 

EQ143 

32 . 3086 

34 . 0024 

.5742 

.  7802 

EQ158 

33.2969 

30 . 7350 

.5951 

.  7745 
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Reliability  Coefficients 


N  of  Cases  =  256.0  N  of  Items  =  10 

Alpha  =  .8058 


RELIABILITY  ANALYSIS  -  SCALE  (SELFMON) 


Mean 

Std  Dev 

Cases 

1 . 

EQ14 

4 . 0938 

.  9241 

256 . 0 

2  . 

EQ32 

4 .1484 

1.0961 

256.0 

3  . 

EQ4  8 

4.2656 

.8539 

256 . 0 

4  . 

EQ64 

4.3711 

.8671 

256.0 

5. 

EQ81 

3.5508 

1.2261 

256 . 0 

6. 

EQ97 

3.7930 

.8901 

256 . 0 

7  . 

EQ112 

2.7305 

1.3112 

256.0 

8  . 

EQ128 

2.6172 

1.1211 

256 . 0 

9. 

EQ144 

2 . 6250 

1 . 0098 

256.0 

10. 

EQ159 

3.5898 

1 . 0663 

256 . 0 

N  of 

Statistics  for 

Mean 

Variance 

Std  Dev 

Variables 

SCALE 

35 . 7852 

21.3380 

4 . 6193 

10 

Item-total 

Statistics 

Scale 

Scale 

Corrected 

Mean 

Variance 

Item- 

Alpha 

if  Item 

if  Item 

Total 

if  Item 

Deleted 

Deleted 

Correlation 

Deleted 

EQ14 

31 . 6914 

17 . 9162 

.3283 

.4901 

EQ32 

31 . 6367 

17 . 1028 

.  3346 

.4828 

EQ4  8 

31 . 5195 

17 . 8898 

.3764 

.4813 

EQ64 

31 .4141 

18 .1651 

.3275 

.4925 

EQ81 

32 . 2344 

18 . 0860 

.1677 

.  5365 

EQ97 

31 . 9922 

19.1843 

.  1746 

.  5285 

EQ112 

33 . 0547 

17 . 1578 

.2266 

.  5188 

EQ128 

33 .1680 

18 . 1560 

.2015 

.  5235 

EQ144 

33 .1602 

19.9546 

.  0403 

.5643 

EQ159 

32 .1953 

18 . 0558 

.2367 

.  5127 
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Reliability  Coefficients 


N  of  Cases  =  256.0  N  of  Items  =  10 

Alpha  =  .5402 


R  E  L 

lABILITY  A 

N  A  L  Y  S  I 

S  -  SC 

ALE  (TASK 

Mean 

Std  Dev 

Cases 

1  . 

EQ16 

3 . 4453 

1.2791 

256.0 

2  . 

EQ33 

3 . 6914 

1.0490 

256.0 

3. 

EQ4  9 

3.6055 

1.1768 

256.0 

4  . 

EQ65 

4.2109 

.7531 

256.0 

5. 

EQ83 

3 . 1094 

1.1864 

256 . 0 

6. 

EQ98 

3 .8672 

.8756 

256 . 0 

7  . 

EQ113 

3 . 1016 

1.0979 

256 . 0 

8  . 

EQ130 

2 . 9648 

1.1794 

256 . 0 

9  . 

EQ145 

2 . 8477 

1 . 1325 

256.0 

10  . 

EQ160 

3.3047 

1 . 1786 

256.0 

11 . 

EQ169 

3.7148 

.9624 

256.0 

12  . 

EQ177 

4.3516 

.8549 

256.0 

13  . 

EQ185 

4 . 1328 

.9362 

256.0 

14  . 

EQ193 

3.2227 

1.2467 

256.0 

15. 

EQ201 

3.5352 

1.2105 

256.0 

N  of 

Statistics  for  Mean 

Variance 

Std  Dev 

Variables 

SCALE  53.1055 

77.5771 

8.8078 

15 

Item-total 

Statistics 

Scale 

Scale 

Corrected 

Mean 

Variance 

Item- 

Alpha 

if  Item 

if  Item 

Total 

if  Item 

Deleted 

Deleted 

Correlation 

Deleted 

EQ16 

49.6602 

66.9782 

.4281 

.  8189 

EQ33 

49.4141 

67.9142 

.4952 

.  8140 

EQ4  9 

49.5000 

66.6431 

.4970 

.  8135 

EQ65 

48 . 8945 

71 . 5457 

.4289 

.  8193 

EQ83 

49.9961 

66.3412 

.5085 

.  8127 

EQ98 

49.2383 

70.1038 

.4574 

.  8170 

EQ113 

50 . 0039 

68.7176 

.4205 

.  8187 

EQ130 

50 . 1406 

66.5056 

.5033 

.  8131 

EQ145 

50.2578 

69.6195 

.3532 

.  8232 

EQ160 

49.8008 

66.6700 

.4945 

.  8137 

EQ169 

49.3906 

71.0390 

.3459 

.  8229 

EQ177 

48 . 7539 

71 . 1353 

.3960 

.  8203 

EQ185 

48 . 9727 

70 . 6463 

.3847 

.  8207 

EQ193 

49.8828 

67.2725 

.4279 

.  8187 

EQ201 

49.5703 

65 . 0539 

.5663 

.8084 
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Reliability  Coefficients 


N  of  Cases  =  256.0  N  of  Items  =  15 

Alpha  =  .8272 


R  E 

L  I  A  B  I  L 

I  T  Y  A 

N  A  L  Y  S  I 

S  -  S  C  A 

L  E  (U  V) 

Mean 

Std  Dev 

Cases 

1  . 

EQ17 

1 . 8242 

1 . 1081 

256.0 

2  . 

EQ34 

3.3398 

1.2671 

256.0 

3  . 

EQ50 

1.7734 

.  9759 

256 . 0 

4  . 

EQ67 

2.7383 

1.2887 

256 . 0 

5. 

EQ84 

2.4688 

1 . 1976 

256.0 

6. 

EQ99 

2.3672 

1 . 1740 

256.0 

7  . 

EQ114 

2 . 9609 

1.2767 

256.0 

8  . 

EQ131 

1.7500 

.  9862 

256 . 0 

9. 

EQ146 

3.1406 

1.2028 

256.0 

10  . 

EQ161 

3.3516 

1.2931 

256 . 0 

N 

of 

Statistics  for 

Mean 

Variance 

Std  Dev  Var 

iables 

SCALE 

25 .7148 

41.2321 

6.4212 

10 

Item-total 

Statistics 

Scale 

Scale 

Corrected 

Mean 

Variance 

Item- 

Alpha 

if  Item 

if  Item 

Total 

if  Item 

Deleted 

Deleted 

Correlation 

Deleted 

EQ17 

23.8906 

36.5292 

.2595 

.7323 

EQ34 

22.3750 

34.7608 

.  3256 

.7245 

EQ50 

23 . 9414 

35.7730 

.  3860 

.  7153 

EQ67 

22 . 9766 

31 . 9210 

.5254 

.  6908 

EQ84 

23 . 2461 

32.7745 

.5122 

.  6944 

EQ99 

23 . 3477 

33.4669 

.  4702 

.  7014 

EQ114 

22 . 7539 

33 . 6608 

.  4011 

.7122 

EQ131 

23 . 9648 

37.3439 

.2419 

.  7331 

EQ146 

22 . 5742 

35.3827 

.  3077 

.  7265 

EQ161 

22 . 3633 

32.2244 

.  4997 

.  6953 
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Reliability  Coefficients 


N  of  Cases  =  256.0 

Alpha  =  .7344 


R  E  L 

lABILITY  A 

N  A  L  Y  S 

Mean 

1  . 

EQ6 

4 . 0898 

2. 

EQ22 

3 . 2266 

3. 

EQ38 

3.4766 

4  . 

EQ54 

3 . 9063 

5  . 

EQ76 

3 . 8789 

6. 

EQIOI 

3.2266 

7  . 

EQ116 

3 . 3320 

8  . 

EQ129 

2 . 9297 

9  . 

EQ163 

3.6328 

10  . 

EQ176 

3.2695 

11 . 

EQ200 

3.6836 

Statistics  for  Mean 

SCALE  38.6523 

Variance 

47.4434 

Item-total 

Statistics 

Scale 

Scale 

Mean 

Variance 

if  Item 

if  Item 

Deleted 

Deleted 

EQ6 

34 .5625 

40.7098 

EQ22 

35 . 4258 

38 . 1435 

EQ38 

35 . 1758 

39.9886 

EQ54 

34 . 7461 

41.0137 

EQ76 

34 . 7734 

41.2896 

EQIOI 

35 . 4258 

39.7984 

EQ116 

35.3203 

37 . 9990 

EQ129 

35.7227 

38 .4679 

EQ163 

35 . 0195 

49.3055 

EQ176 

35 . 3828 

37 . 1862 

EQ200 

34 . 9688 

38 . 8696 

N  of  Items  =  10 


S  -  SCALE  (INTTOL) 


Std  Dev 

Cases 

1.0496 

256.0 

1 . 1663 

256.0 

1.2678 

256.0 

1.0878 

256.0 

1.0833 

256.0 

1.2816 

256.0 

1.2907 

256.0 

1.2025 

256.0 

1.2672 

256.0 

1.2499 

256.0 

1.1970 

256.0 

N  of 

Std  Dev  Variables 
6.8879  11 


Corrected 

Item- 

Alpha 

Total 

if  Item 

Correlation 

Deleted 

.4205 

.  7104 

.5511 

.  6910 

.3647 

.  7173 

.3765 

.  7156 

.3577 

.7180 

.3712 

.  7164 

.  4889 

.  6984 

.5048 

.  6970 

-.1949 

.7917 

.5704 

.  6862 

.4784 

.  7009 
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Reliability  Coefficients 


N  of  Cases  =  256.0  N  of  Items  =  11 

Alpha  =  .7341 


RELIABILITY  ANALYSIS  -  SCALE  (CONSIST) 


Mean 

Std  Dev 

Cases 

1 . 

EQ3 

3.5898 

1 . 1714 

256 . 0 

2  . 

EQ15 

2 . 5156 

1.2648 

256 . 0 

3. 

EQ30 

3 . 7891 

1 . 0858 

256 . 0 

4  . 

EQ42 

4.3320 

.  9877 

256.0 

5  . 

EQ55 

3.2344 

1 .1166 

256 . 0 

6. 

EQ66 

2 . 7227 

1.2196 

256.0 

7  . 

EQ82 

3.7891 

1.2440 

256 . 0 

8  . 

EQ91 

2.6133 

1 . 3844 

256.0 

9. 

EQ136 

2 . 6953 

1.2591 

256 . 0 

10  . 

EQ151 

3.5195 

1 .1946 

256.0 

11 . 

EQ191 

2 .4570 

1.2232 

256 . 0 

N  of 

Statistics  for 

Mean 

Variance 

Std  Dev 

Variables 

SCALE 

35 . 2578 

38 . 8823 

6.2356 

11 

Item-total 

Statistics 

Scale 

Scale 

Corrected 

Mean 

Variance 

Item- 

Alpha 

if  Item 

if  Item 

Total 

if  Item 

Deleted 

Deleted 

Correlation 

Deleted 

EQ3 

31 . 6680 

32.6932 

.  3596 

.  6196 

EQ15 

32 . 7422 

33.7450 

.2407 

.  6424 

EQ30 

31 .4688 

34 . 0853 

.2854 

.  6334 

EQ42 

30 . 9258 

35.7317 

.  1842 

.  6491 

EQ55 

32 . 0234 

31 . 6857 

.4733 

.  5996 

EQ66 

32 . 5352 

33.2144 

.2974 

.  6312 

EQ82 

31 .4688 

32 . 8775 

.3125 

.  6283 

EQ91 

32 . 6445 

34 .1673 

.  1729 

.  6585 

EQ136 

32 .5625 

33 . 1569 

.2855 

.  6336 

EQ151 

31 . 7383 

32.3195 

.  3781 

.  6158 

EQ191 

32 . 8008 

32 . 0974 

.3816 

.  6148 

Reliability 

Coefficients 

N  of  Cases 

256.0 

N  of  Items  =  11 

Alpha  = 

.  6521 

B-18 


Descriptive  Statistics  and  Internal  Consistency  Reliability  for 

Revised  EQ  Scales 


Valid 


Variable 

Mean 

Std  Dev 

Minimum 

Maximum 

N 

RDECIDE 

63.67 

13 . 91 

20.0000 

100.0000 

256 

RINTTOL 

68 . 42 

13 . 34 

23.6364 

100.0000 

257 

RSELFMON 

73 . 63 

9.89 

28.8889 

100.0000 

268 

R  E  L 

I  A  B  I  L  I 

T  Y  A 

N  A  L  Y  S  I 

S  -  S  C  A  L 

E  (D  E  C  I 

Mean 

Std  Dev 

Cases 

1  . 

EQ4 

4 . 0898 

.  8091 

256.0 

2  . 

EQ37 

3.1523 

1.1733 

256 . 0 

3  . 

EQ70 

3.1680 

1 . 1911 

256.0 

4  . 

EQ87 

2.8984 

1.0762 

256.0 

5. 

EQ103 

2 . 9141 

1.2774 

256.0 

6. 

EQ118 

2 . 6875 

1.3180 

256.0 

7  . 

EQ134 

3.7383 

.9527 

256.0 

8. 

EQ149 

3.0977 

1.1420 

256.0 

9. 

EQ164 

3.2773 

1.1603 

256.0 

10. 

EQ171 

3.1211 

1.1869 

256 . 0 

11 . 

EQ180 

3.8047 

.8949 

256 . 0 

12  . 

EQ187 

2.3984 

1.3392 

256.0 

13. 

EQ195 

3 . 0352 

1.2186 

256.0 

N 

of 

Statistics  for 

Mean 

Variance 

Std  Dev  Variables 

SCALE 

41 . 3828 

81 . 7980 

9.0442 

13 

Item-total  Statistics 

Scale 

Scale 

Corrected 

Mean 

Variance 

Item- 

Alpha 

if 

Item 

if  Item 

Total 

if  Item 

Deleted 

Deleted 

Correlation 

Deleted 

EQ4 

37  . 

2930 

76.2315 

.3477 

.  8567 

EQ37 

38  . 

2305 

69.0251 

.5845 

.  8436 

EQ7  0 

38. 

2148 

68.4831 

.  6034 

.  8423 

EQ87 

38. 

4844 

70.9880 

.5322 

.  8471 

EQ103 

38  . 

4688 

65 . 8971 

.  6880 

.  8361 

EQ118 

38  . 

6953 

67 . 4362 

.5832 

.  8436 

EQ134 

37  . 

6445 

74.3869 

.3957 

.  8546 

EQ149 

38  . 

2852 

69.7968 

.5606 

.  8452 

EQ164 

38. 

1055 

71 . 7731 

.4414 

.  8526 

EQ171 

38  . 

2617 

69.5979 

.5449 

.  8462 

EQ180 

37. 

5781 

74 . 4017 

.4272 

.  8530 

EQ187 

38  . 

9844 

70 . 6115 

.4173 

.  8556 

EQ195 

38  . 

3477 

68 . 1728 

.  6033 

.  8423 

B-19 


Reliability  Coefficients 

N  of  Cases  =  256.0  N  of  Items  =  13 

Alpha  =  . 8579 


RELIABILITY  ANALYSIS  -  SCALE  (SELFMON) 


Mean 

Std  Dev 

Cases 

1 . 

EQ14 

4 . 0938 

.  9241 

256.0 

2  . 

EQ32 

4 . 1484 

1 . 0961 

256 . 0 

3  . 

EQ48 

4.2656 

.8539 

256.0 

4  . 

EQ64 

4.3711 

.8671 

256 . 0 

5. 

EQ81 

3.5508 

1.2261 

256.0 

6. 

EQ97 

3.7930 

.8901 

256.0 

7  . 

EQ112 

2.7305 

1.3112 

256.0 

8  . 

EQ128 

2.6172 

1 . 1211 

256.0 

9. 

EQ159 

3.5898 

1.0663 

256.0 

N  of 

Statistics  for 

Mean 

Variance 

Std  Dev 

Variables 

SCALE 

33 .1602 

19.9546 

4.4671 

9 

Item-total 

Statistics 

Scale 

Scale 

Corrected 

Mean 

Variance 

Item- 

Alpha 

if  Item 

if  Item 

Total 

if  Item 

Deleted 

Deleted 

Correlation 

Deleted 

EQ14 

29.0664 

16.5250 

.  3428 

.5141 

EQ32 

29 . 0117 

15.5332 

.3727 

.4995 

EQ48 

28 .8945 

16.4869 

.  3949 

.  5040 

EQ64 

28 . 7891 

16.7161 

.  3507 

.  5143 

EQ81 

29  .  6094 

16.5684 

.1886 

.  5606 

EQ97 

29 . 3672 

17 . 5039 

.2227 

.  5453 

EQ112 

30 . 4297 

16.1519 

.1977 

.  5608 

EQ128 

30 . 5430 

17.2687 

.  1534 

.5679 

EQ159 

29 . 5703 

16.8735 

.2219 

.  5463 

Reliability  Coefficients 

N  of  Cases  =  256.0  N  of  Items  =  9 


Alpha  =  .5643 
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RELIABILITY 


ANALYSIS 


SCALE  (INTTOL) 


Mean  Std  Dev  Cases 


1 . 

EQ6 

4 . 0898 

1.0496 

256.0 

2  . 

EQ22 

3.2266 

1.1663 

256.0 

3. 

EQ38 

3 .4766 

1.2678 

256 . 0 

4  . 

EQ54 

3.9063 

1.0878 

256.0 

5  . 

EQ76 

3.8789 

1 . 0833 

256 . 0 

6. 

EQIOI 

3.2266 

1.2816 

256.0 

7  . 

EQ116 

3.3320 

1.2907 

256.0 

8  . 

EQ129 

2 . 9297 

1.2025 

256.0 

9. 

EQ144 

2 . 6250 

1 . 0098 

256.0 

10  . 

EQ176 

3.2695 

1.2499 

256.0 

11 . 

EQ200 

3.6836 

1.1970 

256.0 

N  of 

Statistics  for 

Mean 

Variance 

Std  Dev 

Variables 

SCALE 

37 . 6445 

53.9633 

7.3460 

11 

Item-total  Statistics 


Scale 

Scale 

Corrected 

Mean 

Variance 

Item- 

Alpha 

if  Item 

if  Item 

Total 

if  Item 

Deleted 

Deleted 

Correlation 

Deleted 

EQ6 

33 . 5547 

46.7264 

.4276 

.7760 

EQ22 

34 . 4180 

44.0638 

.5515 

.7622 

EQ38 

34 .1680 

45 . 6383 

.3922 

.7803 

EQ54 

33 . 7383 

47 . 1822 

.3746 

.7811 

EQ7  6 

33 .7656 

47 . 6233 

.3456 

.7840 

EQIOI 

34 .4180 

45.2638 

.4092 

.7784 

EQ116 

34 . 3125 

43.2902 

.5303 

.7639 

EQ129 

34 . 7148 

44.2203 

.5188 

.7656 

EQ144 

35 . 0195 

49.3055 

.2566 

.7917 

EQ176 

34 . 3750 

42 . 7373 

.5913 

.7567 

EQ200 

33 . 9609 

44.8142 

.4815 

.7699 

Reliability 

Coefficients 

N  of  Cases  = 

256.0 

N  of  Items  =  11 

Alpha  =  .7903 
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Intercorrelation  Matrix  for  Experience  Questionnaire  Scales 
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Factor  Analyses  of  Experience  Questionnaire  Scales 


->  FACTOR 

->  /VARIABLES  tcompose  tconcen  tconsist  tcoop  texecute  tflex  tintense 
self aw 

->  tselfcon  tsustain  ttakechg  ttask  rdecide  rinttol  rselfmon  /MISSING 
->  PAIRWISE  /ANALYSIS  tcompose  tconcen  tconsist  tcoop  texecute  tflex 
tintense 

->  tselfaw  tselfcon  tsustain  ttakechg  ttask  rdecide  rinttol  rselfmon 
->  /PRINT  INITIAL  DET  KMO  EXTRACTION  ROTATION 
->  /FORMAT  SORT  BLANK (.3) 

->  /PLOT  EIGEN 

->  /CRITERIA  FACTORS (2)  ITERATE (125) 

->  /EXTRACTION  PAF 

->  /CRITERIA  ITERATE (125)  DELTA(O) 

->  /ROTATION  OBLIMIN  . 


___________  FACTOR  ANALYSIS  _____ 

Analysis  number  1  Pairwise  deletion  of  cases  with  missing  values 
Determinant  of  Correlation  Matrix  =  .0000012 

Kaiser-Meyer-Olkin  Measure  of  Sampling  Adequacy  =  .95649 

Bartlett  Test  of  Sphericity  =  3406.5362,  Significance  =  .00000 


Extraction  1  for  analysis  1,  Principal  Axis  Factoring  (PAF) 

Initial  Statistics: 


Variable 

Communality 

★ 

★ 

Factor 

Eigenvalue 

Pet  of  Var 

Cum  Pet 

TCOMPOSE 

. 75454 

:*r 

1 

9.56646 

63 . 8 

63 . 8 

TCONCEN 

. 77146 

'k 

2 

1.04694 

7 . 0 

70 . 8 

TCONSIST 

. 55017 

•k 

3 

.85010 

5 . 7 

16. 4 

TCOOP 

. 63265 

k 

4 

.58633 

3 . 9 

80.3 

TEXECUTE 

. 70381 

k 

5 

.46657 

3.1 

83 . 4 

TFLEX 

.  67597 

k 

6 

.40979 

2 . 7 

86.2 

TINTENSE 

. 68912 

k 

7 

.34276 

2.3 

88.5 

TSELFAW 

.46994 

k 

8 

.32151 

2 . 1 

90 . 6 

TSELFCON 

.  73808 

k 

9 

.27277 

1 . 8 

92.4 

TSUSTAIN 

.75997 

k 

10 

.26153 

1.7 

94.2 

TTAKECHG 

.  77090 

k 

11 

.21893 

1 . 5 

95 . 6 

TTASK 

.76046 

★ 

12 

.18967 

1.3 

96.9 

RDECIDE 

.  82259 

k 

13 

.17496 

1.2 

98 . 1 

RINTTOL 

. 56535 

k 

14 

.15521 

1 . 0 

99 . 1 

RSELFMON 

. 48650 

k 

15 

.13647 

.  9 

100.0 
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Hi-Res  Chart  #  2:Factor  scree  plot 

PAF  extracted  2  factors.  9  iterations  required. 

Factor  Matrix: 


Factor  1  Factor  2 


RDECIDE 

.89733 

TTASK 

.  87329 

TCONCEN 

.86419 

TSUSTAIN 

.  84248 

TFLEX 

.  82400 

TSELFCON 

.  82392 

TEXECUTE 

.  82160 

TCOMPOSE 

.81938 

TTAKECHG 

.  81851 

TINTENSE 

.79006 

TCOOP 

.  74150 

.30991 

TCONSIST 

. 68424 

RINTTOL 

. 66287 

.46214 

TSELFAW 

. 64662 

RSELFMON 

.59312 

Final  Statistics: 


Variable 

Communality 

★ 

★ 

Factor 

Eigenvalue  Pet  of 

Var  Cum  Pet 

TCOMPOSE 

. 69381 

★ 

1 

9.25011  61.7 

61 . 7 

TCONCEN 

.75869 

★ 

2 

.68010  4.5 

66.2 

TCONSIST 

. 53827 

* 

TCOOP 

. 64587 

•k 

TEXECUTE 

. 69594 

★ 

TFLEX 

. 68006 

★ 

TINTENSE 

. 69437 

★ 

TSELFAW 

. 45929 

* 

TSELFCON 

. 74490 

k 

TSUSTAIN 

.71079 

k 

TTAKECHG 

. 70327 

★ 

TTASK 

.76266 

* 

RDECIDE 

. 81517 

k 

RINTTOL 

. 65296 

k 

RSELFMON 

.37416 

k 

OBLIMIN 

rotation  1  for 

extraction 

1  in  analysis  1  - 

Kaiser 

Normalization . 
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OBLIMIN  converged  in  8  iterations . 


Pattern  Matrix: 


Factor  1  Factor  2 


TSELFCON 

.  94166 

TINTENSE 

.  92740 

TTAKECHG 

.  83915 

TCOMPOSE 

.79644 

TEXECUTE 

.79134 

RDECIDE 

.78738 

TCONCEN 

.  77521 

TTASK 

.  63123 

TSUSTAIN 

.  57294 

.32349 

TFLEX 

.55795 

.31909 

RINTTOL 

.  89905 

TCOOP 

.  70031 

TCONSIST 

. 61488 

TSELFAW 

. 51137 

RSELFMON 

.41512 

Structure  Matrix: 


Factor  1  Factor  2 


RDECIDE 

.89761 

. 73731 

TCONCEN 

.86723 

.70390 

TSELFCON 

.86006 

. 59709 

TTASK 

.85140 

.76690 

TTAKECHG 

.  83861 

.  62836 

TEXECUTE 

.  83340 

. 64935 

TCOMPOSE 

. 83235 

. 64496 

TINTENSE 

.  82873 

.56362 

TSUSTAIN 

. 81545 

.75300 

TFLEX 

.79716 

.73737 

RINTTOL 

.54675 

.80366 

TCOOP 

.  65655 

.79893 

TCONSIST 

. 61047 

.  72696 

TSELFAW 

.58716 

.  66415 

RSELFMON 

.  54652 

.59153 
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Factor  Correlation  Matrix: 


Factor  1 
Factor  2 


Factor  1 

1 . 00000 
. 74967 


Factor  2 

1 . 00000 


->  FACTOR 

->  /VARIABLES  tcompose  tconcen  tconsist  tcoop  texecute  tflex  tintense 
tselfaw 

->  tselfcon  tsustain  ttakechg  ttask  rdecide  rinttol  rselfmon  /MISSING 
->  PAIRWISE  /ANALYSIS  tcompose  tconcen  tconsist  tcoop  texecute  tflex 
tintense 

->  tselfaw  tselfcon  tsustain  ttakechg  ttask  rdecide  rinttol  rselfmon 
->  /PRINT  INITIAL  DET  KMO  EXTRACTION  ROTATION 
->  /FORMAT  SORT  BLANK (.3) 

->  /PLOT  EIGEN 

->  /CRITERIA  FACTORS (3)  ITERATE (125) 

->  /EXTRACTION  PAF 

->  /CRITERIA  ITERATE (125)  DELTA (0) 

->  /ROTATION  OBLIMIN  , 


FACTOR  ANALYSIS 


Analysis  number  1  Pairwise  deletion  of  cases  with  missing  values 
Determinant  of  Correlation  Matrix  =  .0000012 

Kaiser-Meyer-Olkin  Measure  of  Sampling  Adequacy  =  .95649 

Bartlett  Test  of  Sphericity  =  3406.5362,  Significance  =  .00000 

Extraction  1  for  analysis  1,  Principal  Axis  Factoring  (PAF) 
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Initial  Statistics; 


Variable 

Communality 

"k 

Factor 

Eigenvalue 

Pet  of  Var 

Cum  Pet 

TCOMPOSE 

. 75454 

k 

1 

9.56646 

63 . 8 

63.8 

TCONCEN 

. 77146 

★ 

2 

1.04694 

7.0 

70.8 

TCONSIST 

.55017 

★ 

3 

.85010 

5 . 7 

76 . 4 

TCOOP 

.  63265 

★ 

4 

.58633 

3 . 9 

80.3 

TEXECUTE 

.70381 

* 

5 

.46657 

3.1 

83 . 4 

TFLEX 

. 67597 

★ 

6 

.40979 

2.7 

86.2 

TINTENSE 

. 68912 

k 

7 

.34276 

2.3 

88.5 

T SELF AW 

.46994 

k 

8 

.32151 

2 . 1 

90  .  6 

TSELFCON 

.  73808 

k 

9 

.27277 

1.8 

92.4 

TSUSTAIN 

.75997 

k 

10 

.26153 

1 . 7 

94.2 

TTAKECHG 

.77090 

k 

11 

.21893 

1.5 

95 . 6 

TTASK 

.76046 

k 

12 

.18967 

1.3 

96.9 

RDECIDE 

.  82259 

k 

13 

.17496 

1.2 

98 . 1 

RINTTOL 

.56535 

k 

14 

.15521 

1 .  0 

99 . 1 

RSELFMON 

.  48650 

k 

15 

.  13647 

.  9 

100 . 0 

Hi-Res  Chart  #  3: Factor  scree  plot 

PAF  extracted  3  factors.  10  iterations  required. 

Factor  Matrix; 


Factor  1  Factor  2 


Factor  3 


RDECIDE 

.89940 

TTASK 

.  87200 

TCONCEN 

.86400 

TSUSTAIN 

.  84689 

TTAKECHG 

. 83434 

TCOMPOSE 

.  82325 

TFLEX 

.  82227 

TSELFCON 

.  82194 

TEXECUTE 

.  82151 

TINTENSE 

.78770 

TCOOP 

.74540 

TCONSIST 

.  68470 

RINTTOL 

. 66152 

TSELFAW 

. 64515 

RSELFMON 

.59921 

.38325 

.33041 
.  45897 

.31765 
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Final  Statistics: 


Variable 

Communality 

★ 

■k 

Factor 

Eigenvalue 

Pet  of  Var 

Cum  Pet 

TCOMPOSE 

.  75995 

•k 

1 

9.29123 

61 .  9 

61  .  9 

TCONCEN 

.  77746 

k 

2 

.71316 

4  .  8 

66.7 

TCONSIST 

.56717 

k 

3 

. 54771 

3 . 7 

70 . 3 

TCOOP 

.71454 

k 

TEXECUTE 

.71386 

k 

TFLEX 

.  68149 

k 

TINTENSE 

. 68610 

★ 

TSELFAW 

.  45787 

★ 

TSELFCON 

.  74203 

★ 

TSUSTAIN 

.78191 

k 

TTAKECHG 

. 90028 

k 

TTASK 

.76890 

k 

RDECIDE 

. 85826 

k 

RINTTOL 

.  65536 

k 

RSELFMON 

.48692 

k 

OBLIMIN  rotation  1  for  extraction  1  in  analysis  1  -  Kaiser 
Normalization . 


OBLIMIN  converged  in  27  iterations. 


Pattern  Matrix: 


Factor  1  Factor  2 


Factor  3 


TCOMPOSE 

.  92410 

RDECIDE 

.88793 

TINTENSE 

.  88396 

TSELFCON 

.  84504 

TCONCEN 

.  83126 

TSUSTAIN 

.  71877 

TEXECUTE 

.  67562 

TTAKECHG 

.  63278 

.  47775 

TFLEX 

.  58946 

TTASK 

.56250 

. 34973 

TCOOP 

.  82284 

RINTTOL 

.  80036 

RSELFMON 

.59971 

TCONSIST 

.52205 

TSELFAW 

.51591 
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structure  Matrix: 


Factor  1 

Factor  2 

Factor 

RDECIDE 

.91781 

.71443 

TCONCEN 

. 87781 

. 68925 

TCOMPOSE 

. 85913 

. 61683 

TSUSTAIN 

. 84471 

.72682 

TSELFCON 

. 84340 

.  61124 

TTASK 

.83833 

.77763 

TINTENSE 

. 82115 

.  56716 

TTAKECHG 

. 81681 

.  67172 

.56779 

TEXECUTE 

. 81416 

.66936 

.30196 

TFLEX 

. 80111 

.72762 

TCOOP 

. 63605 

.  83617 

RINTTOL 

.56350 

.78046 

TCONSIST 

. 62968 

.  70648 

TSELFAW 

.  58318 

. 66342 

RSELFMON 

.  51575 

. 64402 

.31080 

Factor 

Correlation  Matrix: 

Factor  1 

Factor  2 

Factor  3 

Factor 

1 

1 . 00000 

Factor 

2 

.74630 

1 . 00000 

Factor 

3 

.  12434 

. 06801 

1.00000 

->  FACTOR 

->  /VARIABLES  tcompose  tconcen  tconsist  tcoop  texecute  tflex  tintense 
tselfaw 

->  tselfcon  tsustain  ttakechg  ttask  rdecide  rinttol  rselfmon  /MISSING 
->  PAIRWISE  /ANALYSIS  tcompose  tconcen  tconsist  tcoop  texecute  tflex 
tintense 

->  tselfaw  tselfcon  tsustain  ttakechg  ttask  rdecide  rinttol  rselfmon 
->  /PRINT  INITIAL  DET  KMO  EXTRACTION  ROTATION 
->  /FORMAT  SORT  BLANK (.3) 

->  /PLOT  EIGEN 

->  /CRITERIA  FACTORS (4)  ITERATE (125) 

->  /EXTRACTION  PAF 

->  /CRITERIA  ITERATE (125)  DELTA(O) 

->  /ROTATION  OBLIMIN  . 
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______  FACTOR  ANALYSIS  __________ 

Analysis  number  1  Pairwise  deletion  of  cases  with  missing  values 
Determinant  of  Correlation  Matrix  =  .0000012 


Kaiser-Meyer-Olkin  Measure  of  Sampling  Adequacy  =  .95649 


Bartlett  Test  of  Sphericity  =  3406.5362,  Significance  =  .00000 


Extraction  1  for  analysis  1,  Principal  Axis  Factoring  (PAF) 


Initial  Statistics: 


Variable 

Communality 

★ 

■k 

Factor 

Eigenvalue 

Pet  of  Var 

Cum  Pet 

TCOMPOSE 

.  75454 

•k 

1 

9.56646 

63 . 8 

63.8 

TCONCEN 

,  77146 

★ 

2 

1 . 04694 

7 . 0 

70.8 

TCONSIST 

,  55017 

★ 

3 

.85010 

5.7 

76.4 

TCOOP 

.  63265 

★ 

4 

. 58633 

3 . 9 

80.3 

TEXECUTE 

.70381 

★ 

5 

.46657 

3 . 1 

83.4 

TFLEX 

.  67597 

k 

6 

.40979 

2 . 7 

86.2 

TINTENSE 

.  68912 

k 

7 

.  34276 

2 . 3 

88.5 

TSELFAW 

.46994 

k 

8 

.  32151 

2 . 1 

90 . 6 

TSELFCON 

. 73808 

k 

9 

.27277 

1 .  8 

92.4 

TSUSTAIN 

.75997 

k 

10 

.26153 

1  .  7 

94.2 

TTAKECHG 

.  77090 

k 

11 

.21893 

1 . 5 

95 . 6 

TTASK 

.76046 

k 

12 

.18967 

1 . 3 

96.9 

RDECIDE 

.  82259 

k 

13 

.  17496 

1 . 2 

98 . 1 

RINTTOL 

.56535 

k 

14 

. 15521 

1 . 0 

99 . 1 

RSELFMON 

.48650 

k 

15 

.  13647 

.  9 

100 . 0 

Hi-Res  Chart  #  4:Factor  scree  plot 

PAF  extracted  4  factors.  14  iterations  required. 
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Factor  Matrix: 


RDECIDE 

TTASK 

TCONCEN 

TSUSTAIN 

TTAKECHG 

TCOMPOSE 

TSELFCON 

TFLEX 

TEXECUTE 

TINTENSE 

TCOOP 

TCONSIST 

RINTTOL 

T SELF AW 

RSELFMON 


Factor  1 

.  89836 
.  87413 
.86494 
.  84610 
. 83184 
.  82478 
.  82461 
.  82182 
.  82077 
.79050 
.  74327 
. 68616 
. 66215 
.  64594 
.  60581 


Factor  2 


.32787 
.  45878 


Factor  3 


.37582 


.36980 


Factor  4 


Final  Statistics: 


Variable 

Communality 

* 

* 

Factor 

Eigenvalue 

Pet  of  Var 

Cum  Pet 

TCOMPOSE 

.  78568 

•k 

1 

9.30726 

62.0 

62.0 

TCONCEN 

.79554 

★ 

2 

.73142 

4  .  9 

66.9 

TCONSIST 

.59563 

★ 

3 

.56981 

3.8 

70.7 

TCOOP 

.  69516 

k 

4 

.23138 

1 . 5 

72.3 

TEXECUTE 

.  71375 

•k 

TFLEX 

.  68427 

■k 

TINTENSE 

.  72676 

k 

TSELFAW 

.  47742 

k 

TSELFCON 

.  78081 

k 

TSUSTAIN 

.78130 

k 

TTAKECHG 

.  87916 

k 

TTASK 

.79966 

k 

RDECIDE 

.  85573 

k 

RINTTOL 

.  67184 

k 

RSELFMON 

.59719 

★ 

OBLIMIN  rotation  1  for  extraction  1  in  analysis  1  -  Kaiser 
Normalization . 


OBLIMIN  converged  in  31  iterations. 


B-31 


Pattern  Matrix: 


Factor  1 

Factor  2 

Factor  3 

Factor 

TCOMPOSE 

. 90971 

TINTENSE 

.  80343 

TCONCEN 

.  78682 

RDECIDE 

.  68877 

TSUSTAIN 

.  59239 

.38539 

TELEX 

.  52585 

TCONSIST 

.  61410 

RINTTOL 

.56328 

TSELFAW 

.45242 

RSELFMON 

.70927 

TCOOP 

.  44203 

.45586 

TTAKECHG 

.38389 

-  .  55778 

TSELFCON 

.40049 

-.52688 

TTASK 

.33962 

-.43138 

TEXECUTE 

.36992 

- . 38061 

Structure 

Matrix : 

Factor  1 

Factor  2 

Factor  3 

Factor 

RDECIDE 

.  90320 

.  67486 

.49811 

- . 65378 

TCONCEN 

.88599 

.59701 

.  55904 

- . 58392 

TCOMPOSE 

.  88488 

.56083 

.  45522 

-.54085 

TSUSTAIN 

.  83493 

.73209 

.46465 

- . 54422 

TINTENSE 

.  83023 

.41338 

.53624 

- . 61757 

TFLEX 

.  78934 

.  64154 

.58388 

- . 54507 

TTASK 

.  77471 

.  67461 

.  62255 

- . 73822 

TEXECUTE 

.76424 

.51451 

. 60360 

- . 72908 

RINTTOL 

.56540 

.76517 

.58556 

TCONSIST 

.  60521 

.74706 

.43300 

- . 38204 

TSELFAW 

.  53427 

.  61630 

.50200 

- .46143 

RSELFMON 

.48047 

.42589 

.76869 

- .39986 

TCOOP 

.  58676 

. 69618 

.72812 

-  .  48117 

TTAKECHG 

.  73882 

.41794 

.71835 

- . 84994 

TSELFCON 

.  78848 

.49273 

.49532 

- . 80846 

Factor  Correlation  Matrix: 

Factor  1 

Factor  2 

Factor  3 

Factor  4 

Factor  1 

1 . 00000 

Factor  2 

.  61439 

1 . 00000 

Factor  3 

.  54467 

.49582 

1 . 00000 

Factor  4 

-  .  64318 

-.31241 

-.45668 

1 . 00000 

B-32 


